Multi-user virtual and augmented reality tracking systems

ABSTRACT

Systems and methods are described for virtual and augmented reality. A position of a first user and a position of a second user in a physical space are received, wherein the positions are derived from detected infrared light from a plurality of cameras. An image corresponding to the first user is rendered at a first virtual position in a display device associated with the second user, wherein the first virtual position is determined based at least in part on a position of the first user in the physical space and on a determined viewpoint of the second user. An image corresponding to the second user is rendered at a second virtual position in a display device associated with the first user, wherein the second virtual position is determined based at least in part on a position of the second user in the physical space and on a determined viewpoint of the first user.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication, are hereby incorporated by reference in their entiretyunder 37 CFR 1.57.

COPYRIGHT RIGHTS

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the reproduction by any one of the patent document or thepatent disclosure, as it appears in the patent and trademark officepatent file or records, but otherwise reserves all copyright rightswhatsoever.

FIELD OF THE INVENTION

The present invention relates to systems and methods for determining aposition and identity of infrared tracking markers used in associationwith virtual reality and/or augmented reality simulations.

BACKGROUND

Virtual Reality (VR) comprises a computer simulated environment that cansimulate a physical presence in places in the real world or imaginedworlds. Conventionally, virtual reality environments are primarilyvisual experiences, displayed on a screen (e.g., and viewed by a userusing 3D glasses) or through special stereoscopic display head gear. Thesimulated environments can be similar to the real world in order tocreate lifelike experiences or it differs significantly from reality,such as in VR games.

Augmented Reality (AR) generally refers to a computer simulatedenvironment combined with the real world. Conventionally, the elementsof the real world are augmented with computer generated graphics. Often,translucent stereoscopic headsets are worn by the user in AR simulationsto enable a wearer to view the real world through the headset while alsobeing able to view computer generated graphics.

Movement of participants and/or objects in interactive VR and ARsimulations are optionally tracked using various methods and devices.

Multi-user environments greatly expand the potential of virtualenvironments by allowing users to inhabit the same virtual world. Assuch, people can socialize, negotiate, play games, work together orotherwise interact in an entirely digital context. Moreover, a user canbe in a completely different part of the planet and still experience the“physical” presence of another participant.

SUMMARY OF THE INVENTION

An aspect of the disclosure relates to an example marker identificationand position tracking system configured to interface and work inconjunction with a marker device and camera system and to provide highfidelity tracking of user and object motion in a virtual and/oraugmented reality experience.

An aspect of the disclosure relates to an example computing systemconfigured to interface and work in conjunction with a tracking systemand/or certain sensors, to provide a high quality, interactive,multi-user, virtual reality, and/or augmented reality experience.Optionally, the example computing system enables use case scenarios inwhich certain detected tracking and/or sensor data is provided to anoperator/user and not provided to or restricted to other participants ofthe virtual and/or augmented reality experience. Optionally, the receiptof additional or enhanced detected tracking and/or sensor dataoptionally informs the operator/user regarding a one or moreparticipant's experience during or subsequent to the session.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example operating environment.

FIG. 2 illustrates an example operating environment of a precisionpositioning tracking motion capture system.

FIG. 3 illustrates a process of an example embodiment of an enhancedtracking virtual reality system.

FIG. 4 and FIG. 5 illustrate an example process.

FIG. 6 illustrates an example virtual reality scenario.

FIG. 7 illustrates an example operating environment of a clusteredsystem with a master server and multiple clients.

FIG. 8 illustrates an example operating environment of a precisionpositioning tracking motion capture system of a clustered system.

FIGS. 9 and 10 illustrate a process of a multi-user virtual realityexample embodiment.

FIG. 11 illustrates an example configuration of software modules of aclustered system.

DETAILED DESCRIPTION

An aspect of this disclosure relates generally to devices and methodsfor enabling an interactive, multi-user virtual or augmented realityexperience, and in particular, to a computing system which works inconjunction with sensory detection and motion capture and tracking inputdevices.

Other example embodiments of a multi-user virtual or augmented realityexperience that may be utilized with the systems and processed describedherein are described in U.S. patent application Ser. No. 14/971,720,titled “MULTI-USER VIRTUAL REALITY PROCESSING”, the contents of whichare incorporated herein in their entirety by reference.

Advantageously, virtual/augmented reality and/or simulated environmentsas described herein can optionally be used in a number of applicationareas including but not limited to: real estate, architecture, productdesign, human behavior analysis, user training, gaming, product andmarketing evaluation, design verification, data visualization,teleoperation, and telepresence or any physical environment that forreasons of, for example, cost or potential harm to a user can besimulated rather than directly experienced. In addition, virtual realityas described herein may also be used in augmented reality environments,including for examples, in an automotive, construction, and assemblyvirtual reality environments.

Recently, multi-user VR environments/worlds have been created whichenable users to collaborate and share a virtual world experience. Motiontracking is a crucial component of most virtual reality systems. Bytracking the position and orientation of each user, the computing systemcan control the simulation's viewpoint such that the user may navigatethrough the virtual world just as naturally as they would navigatethrough the physical world. In addition, tracked movements of a user inthe physical world may be mapped onto certain movements of an avatarrepresentation of the user in the virtual/augmented world as viewed byone or more participants/operator of the simulation session. Thus, in amulti-user virtual/augmented reality session, each user has a view ofthe virtual/augmented world/scene (from the user's unique viewpointand/or position in the virtual/augmented reality world/scene) and one ormore avatars representing each session participants in the scene.Optionally, the computing system further enables one or moreoperators/observers to view a multi-user virtual/augmented realitysession via one or more monitors from one or more positions in thevirtual/augmented reality world. Optionally, a one or moreoperators/observers may observe the virtual/augmented reality sessionfrom the viewpoint of one or more of the session participants.Conventionally, a session participant's movement may be tracked, forexample, via motion tracking cameras and tracking devices (e.g., viatracking devices affixed to a user) or other means (e.g., haptic gloves,etc.). Optionally, virtual world sound effects and each user's voice arebroadcast to other session participants and/or sessionobservers/operators. Optionally, other perceptional senses may beprovided to other session participants and/or sessionobservers/operators including the sense of smell, touch, and taste toenable a fully immersive and interactive experience.

Certain marker identification systems used in virtual and augmentedreality tracking systems use infrared light. Advantageously, infraredlight is invisible to human eyes and, thus, is not distracting tosimulation participants and/or operators. Further, infrared light isrelatively easy to record and the emitted and/or reflected light can bedistinguished (e.g., via image processing) in recorded images. However,other wavelengths/colors and light types may be used. Conventionally,infrared marker identification systems used in virtual and augmentedreality environments use reflective or active markers.

With respect to the use of reflective markers, a given camera may havean infrared light emitting strobe attached to or integrated with aninfrared sensitive camera. The infrared light emitted from the strobereflects off one or markers attached to or associated with a subject orobject and is received at the camera.

To track an object, markers may be placed (e.g., with adhesive, clip,tape, magnets, snaps, sewn on, or otherwise placed/affixed) on an objectto be tracked. For example, three markers or more markers may beutilized in order to track an object in 6 degrees of freedom, x-axis,y-axis, z-axis, pitch, yaw, roll. If more than one subject (wherein thesubject may be a moveable object) is to be tracked in a given trackingarea, the plurality of markers may be placed in unique patterns on thesubject, if each marker in a set of markers is not by itselfdistinguishable from the rest of the markers. A unique reflected patternrecorded in the video images enables the system via certain calculationsand/or algorithms to determine at a given moment (e.g., video frame orset of frames) the location and/or orientation of the subject to whichthe markers are attached.

With respect to the active marker system, pulsed light (e.g., infraredlight) is emitted from a given active marker as to provide tracking andidentification information. Advantageously, a given marker in a trackingfield may optionally be configured to emit a unique identifier whereinthe identifier is defined by a unique pattern of pulsed light. Thus, forexample, an active marker using an 8-bit binary encoding can encode 256markers, a 5-bit binary encoding can encode 32 markers, etc. Otheractive marker systems may also be used, including, for example, a markersystem in which only a single marker in a set of markers is emittinglight at any given time period (e.g., a video frame or a set of frames).This optional active marker systems may have reduced tracking smoothnessrelative to the active marker with integrated marker identification, asthe camera system may need to record each marker binary encoded pattern(e.g., one marker identification per frame or frame set) in the systemof markers before returning to the first marker.)

While both an infrared light reflecting system of markers and aninfrared light emitting system of markers can be used in certainvirtual/augmented reality simulations, an infrared light emittingtracking system may have certain advantageous over a reflective systemwhen tracking multiple subjects in a tracking area. For example, thereflective systems may need to track a larger number of markers and mayneed to further track patterns, increasing the likelihood of two or moremarker patterns traversing across each other when the markers are inclose proximity to each other causing marker confusion (e.g., markerswapping). In addition, in a reflective marker system, as the number ofsubject/objects in a tracking area increase, the number of markersneeded to create distinguishable patterns may increase. Having severalmarkers in a cluster in close proximity to each other can decreasetracking quality as the tracking cameras may not see each marker clearlyin a cluttered marker environment.

An aspect of this disclosure relates generally to devices and methodsfor enabling one or more virtual and/or augmented reality sessionusers/operators to receive certain additional or enhanced sensor and/orposition tracking data associated with a participant not availableand/or disclosed to other participants/operators/observers of thesimulation. Optionally, the computing system applies certain dataanalysis techniques to the enhanced sensor and/or position tracking datain order to derive certain inferences. Optionally, this additionalcomputational processing and data enables a user, operator, and/orobserver to draw certain conclusions regarding the simulation sessionand/or to alter aspects of the session (e.g., a presentation, training,simulation scenes, etc.) during or subsequent to the session. Thisenhanced and/or additional displayed and recorded data are nowdescribed. For clarity, the description herein may refer to virtualreality, which is intended to encompass both virtual reality andaugmented reality, unless the context indicates otherwise. Similarly,the description herein may refer to an operator, which is intended toencompass either a system operator or system administrator responsiblefor among other things configuring and initiating a simulation session,unless the context indicates otherwise.

FIG. 1 illustrates an example system architecture. As described herein,the illustrated system architecture can be used to facilitate a virtualand/or augmented reality experience, and in particular, a multi-uservirtual and/or augmented reality experience with enhanced sensortracking. The system may be configured to perform some or all of thefollowing: image capture and tracking functions (including enhancedtracking as further described herein), video/image processing, speechprocessing, pattern matching, data mining, marker identification (e.g.,LED marker identification), user/object position calculations, actionsynchronization, and/or dynamic rendering of virtual worlds in anoptimized (e.g., in substantially real-time and/or minimal renderingdelays) manner to create a lifelike experience.

As illustrated in FIG. 1, a master server computing device 200 (e.g., ageneral purpose computer, a tablet computer, a cloud-based server, asmart phone, a graphics processing unit, etc.) is coupled to a pluralityof client computing devices 210, 220 over a data network 250 (e.g., alocal area network, a wide-area network, the Internet, a privatenetwork, a public network, etc.). Optionally, the master server 200 maybe coupled to a one or more local participant sensors 110 (e.g., awearable heart rate monitor), cameras 100, and/or microphones 120 (asfurther described herein) which enables said local participant deviceoutputs to be received by the master server 200. Optionally, the variouslocal participant sensor, camera, and microphone outputs are connectedwirelessly or via direct cabling to the master server 200. Optionally,the local participant sensor 110, camera 100, and/or microphone 120interface (e.g., wirelessly) to a local computing device 130 of thelocal participant (e.g., a smartwatch, smart phone, tablet computingdevice, etc.). Optionally, the local participant rawsensor/camera/microphone data is sent to the master server 200.Optionally, the local participant computing device 130 performs certainlocal computational operations with respect to the receivedsensor/camera/microphone data prior to sending the raw data and/or localprocessed output (e.g., a determined heart rate, a determined positionalgaze, etc.) Optionally, the master server 200 may be directly connectedto a console/monitor 260 (or over the data network 250 to aconsole/monitor 260) which displays a user interface via master serversoftware 200 for a user/participant or operator to provision, designate,and/or configure the master server 200 (e.g., download and/or upgradesoftware, provision data communication interfaces, configure accounts,administer security, edit files, etc.). Optionally, the master server200 also provides a computing platform for compiling and/or executingcertain programs and/or programmatic scripts (e.g., simulations),receiving and sending certain communications, performing video/imageprocessing tasks, performing math computations, displaying/renderingcertain images (e.g., rendering virtual worlds), providing clientsynchronization, hosting a client application, etc.

A given computing device 200, 210, and 220 may be further configuredwith a data store 202, 212, and 222, respectively, enabling the serverto store in memory data associated with the VR session, simulations,models, objects, images, certain parameters, executable scripts/code,local event logs, error logs, sensor data, speech, video, etc. Thus, forexample, the master server computing device 200 may host a master serversoftware program 205, as illustrated in FIG. 2, comprising a singlesoftware program or a plurality of software programs or software modulesincluding, for example, a render engine 610 configured to render and/orenable the rendering of VR scenes, a physics engine 615 (e.g., thatprovides a simulation of physical systems, such as rigid and soft bodydynamics, collision detection, and fluid dynamics, and that provides aninterface that hides the low-level details of the physics needed invirtual reality applications to enable application/game developers tofocus on the higher-level functionalities of the application), a rulesengine 620, a simulation control engine 625 (that coordinates simulationexecution), a session manager 630, a simulation state synchronizerengine 635 (that, for example, synchronizes associated clientviewpoints) and/or an error handling 640, a client-server communicationsmanager 650 (that, for example, manages client server communicationsincluding over a data communication network (e.g., a low latency datacommunication network)), resource manager 655 (that, for example,manages resources, including shared resources (e.g., simulation objects,scenes, etc.), speech-to-text conversion module 670, sensor analyticsand inference module 675, video compression module 680, virtual realitytracking and marker identification software 660 (e.g., the Vizard VR™toolkit and PPT Studio software from WorldViz LLC of Santa Barbara) byway of example, enhanced tracking sensor output processing and display670, enhanced tracking analytics 675.

The master server computing device 200 may include cluster nodes (e.g.,companion computing devices) that handle gameplay/simulation logic,tracking software, and one or more additional servers that processcommunications from client computing device. The master server computingdevice 200 may include a login server, including a client/user identityand access authentication engine, that manages login sessions withclient computing devices, validates client computing devices (e.g., bychecking password and device identifiers), and if the client computingdevices are validated, enables the devices to access the mastercomputing device 200 and/or cluster nodes. The data store 202 associatedwith the master server computing device 200 may store user accountinformation such as password information, user/device identifiers,status of game/simulation play which may be stored upon a usersuspending a game/simulation or other virtual/augmented reality activityand which may be accessed and utilized when the activity is restarted toprovide a continuous experience, user usage information (e.g., time andduration of virtual/augmented reality activity), other users the usermay be linked to (e.g., other users the user has interacted with whenengaging in virtual/augmented reality activities), etc.

The master server computing device 200 may further comprise multipleservers distributed over a broad geographic area to reduce latency withrespect to interactions with client server devices, wherein a givenserver (or servers) may serve client computing devices in itsgeographical area. The various master server computing devices mayinclude network interfaces and may be interconnected via high speed datanetworks to further reduce latencies. The cluster nodes may includededicated high speed graphic processors to process virtual/augmentedreality graphic tasks. The master server computing device 200 mayinclude one or more administrator terminals.

Similarly, as illustrated in FIG. 1, one or more client computingdevices 210, 220 (e.g., a general purpose computer, a tablet computer, acloud-based server, a smart phone, a graphics processing unit, a gameconsole, etc.) are coupled to the master server 200 over a data network250 (e.g., a local area network, a wide-area network, the Internet, aprivate network, a public network, etc.). Optionally, each client device210, 220 can connect with other client devices via the data network 250(e.g., in a peer-to-peer manner). Optionally, a given client computingdevice 210, 220 may be connected directly or over a network to a consolewhich displays a user interface via client software 215, 225 to auser/participant or operator and which can be used to provision,designate, and configure the given client computing device 210, 220(e.g., download and/or upgrade software, provision data communicationinterfaces, configure accounts, administer security, edit files, etc.).Optionally, each client device 210, 220 may be coupled to a one or morelocal participant sensors 110 (e.g., a wearable heart rate monitor),cameras 100, and/or microphones 120 (as further described herein) whichenables said local participant device outputs to be received by eachclient device 210, 220. Optionally, the various local participantsensor, camera, and microphone outputs are connected wirelessly or viadirect cabling to each client device 210, 220. Optionally, the localparticipant sensor 110, camera 100, and/or microphone 120 interface(e.g., wirelessly) to a local computing device 130 of the localparticipant (e.g., a smartwatch, smart phone, tablet computing device,etc.). Optionally, the local participant raw sensor/camera/microphonedata is sent to the master server 200. Optionally, the local participantcomputing device 130 performs certain local computational operationswith respect to the received sensor/camera/microphone data prior tosending the raw data and/or local processed output (e.g., a determinedheart rate, a determined positional gaze, etc.) Optionally, enhancedtracking sensor/camera/microphone data (either raw data and/or locallyprocessed output) of a local participant may be provided to the masterserver 200 via the LAN/WAN 250. Optionally, a given client computingdevice 210, 220 provides a computing platform for compiling and/orexecuting certain programs and/or programmatic scripts, receiving andsending certain communications, performing video/image processing tasks,performing speech processing including speech-to-text, performingsensor/camera/microphone output software analytics, identifying distinctmarkers from recorded infrared light, calculating marker positions inspace, performing math computations, enabling the displaying/renderingof certain images (e.g., rendering virtual worlds or augmented images),etc. For example, a given client computing device 210, 220 may include a3D rendering engine, a 3D sound engine, a virtual reality executionengine, an avatar engine, a simulation state synchronizer engine, asession manager engine and/or a network interface. Optionally, theclient computing device 210, 220 may access one or more external datastores 290 (e.g., a cloud-based data store, a network-based server,etc.) over the LAN/WAN 250.

FIG. 1 also illustrates an optional element of the cluster system, aplurality of motion capture cameras 310, 320, 330, and 340. Motiontracking is a crucial component of most virtual reality systems. Bytracking the position and orientation of the user, the cluster systemcan control the simulation's viewpoint such that the user can navigatethrough the virtual world just as naturally as they would navigatethrough the physical world. In addition, tracked movements of anindividual in the physical world may be mapped onto certain movements ofan avatar in the virtual world. Optionally, these motion capture cameras310-340 are connected physically via cabling or wirelessly to the datanetwork 250 and/or directly to the master server 200, client 210, and/orclient 220. Optionally, the video output and/or detected markeridentities and marker positions are streamed from the cameras 310-340 tothe master server computing device 200 and/or the client computingdevices 210,220. Optionally, a plurality of motion capture cameras arepositioned above and/or reside within a staging area in which VR sessionusers participate in a simulation. FIG. 3 illustrates an example camerapositioning in a simulation. By way of example, multiple (e.g., 2, 4, 6,8, 9, 10, 11, 12) tracking cameras (e.g., precision position tracking(PPT) cameras, such as the PPT X™ precision motion tracking cameras fromWorldViz LLC of Santa Barbara) may stream live video to a master servercomputing device 200, or a client computing device 210, 220, which areprovisioned with virtual reality tracking software (e.g., the Vizard VR™toolkit or PPT Studio from WorldViz LLC of Santa Barbara; dll forWindows; C source library for Linux) capable of image processing livevideo images in substantially real-time.

Optionally in addition or instead, a dedicated computing device 230(e.g., a laptop, PC, smartphone, server, etc.) may be optionallyprovisioned with virtual reality software (e.g., Vizard VR and/or PPTStudio from WorldViz LLC of Santa Barbara) capable of processingreceived live camera output stream in substantially real-time.Optionally, the motion capture cameras record image data includingemitted infrared light in a tracking area. Optionally, the recordedimage data is provided (e.g., streamed) to the computing device 230 orother computing device via a data network (e.g., 250). Optionally, thecomputing device receiving the image data has certain software (e.g.,PPT Studio software from WorldViz LLC of Santa Barbara) capable ofdetermining from the image data, a position and identity for eachinfrared marker in the field of view/tracking area of the motion capturecameras.

Optionally, the calculated/determined/derived marker position andidentity information (e.g., tracking data) may be used (e.g., by thecomputing device) to render a change in a scene or orientation of ascene as viewed by a user (e.g., in a head mounted display or otherdisplay). For example, infrared markers may comprise light emittingdevices (LED) which may be configured to be attached to or mounted on anitem worn by the user (e.g., headgear). Optionally, a marker device canbe attached to articles of clothing and/or strapped or otherwiseattached to or physically associated with a user. Optionally, aninfrared marker tracking device includes two or more light emittingcomponents attached to one or more items worn by the user and the systemtracks the light source in three dimensions to determine the user'smovements in time. Optionally, head movement tracking data is collectedin substantially real-time and can be used, for example, in a virtualreality display in which some or all of the displayed images aremodified coincident with the tracked movement of the user. For example,if the tracking system detects sideways movement of the user, theperspective and/or the displayed image viewed by the user may bemodified to simulate the sideways movement of the user (e.g., displayingan image behind a tree which was obstructing the user's view prior tothe user's sideways movement).

Virtual Reality software may include a hardware integration module 660which may incorporate a visual tool for configuring devices that the VRsoftware supports, including displays (e.g., head-mounted displays,multi-screen projection walls, consumer 3D monitors), trackers (headtrackers, gloves, full body motion capture), and input devices (e.g.,wands, steering wheels, gamepads, joysticks, etc.)), feedback devices(e.g., haptic feedback devices that simulate the sensation of force,pressure and/or resistance by using electric actuators, pneumaticshydraulics, and/or neuromuscular stimulators).

The VR software may enable editing of transform nodes (e.g., position,rotation, scale), clone/copy nodes, rename nodes, delete nodes, insertnew group/transform nodes, and add/remove/modify descriptors. The VRrendering software 610 may enable the layout of a virtual scene bycombining and arranging multiple independent 3D models. The VR softwaremay enable a given virtual reality scene or objects to react to a user'snatural body motions. Thus, for example, the VR software may enable auser to interact with head, hands, and feet with virtual objects in amanner similar to real objects to provide a convincing sensation oftelepresence. The VR software optionally provides full body motioncapture technology that delivers full human presence for both firstperson point of view and third person points of view experiences. The VRsoftware optionally enables the integration of live full featureavatars.

An optional aspect of an interactive, multi-user, virtual realitysession is the utilization of one or more motion tracking cameras thatrecord emitted infrared light pulses from tracking emitter devices.Motion tracking camera video output may be received by the computingdevices of the system directly or via various wired or wireless networksor peripheral interfaces/buses. Optionally, motion tracking camera videooutput may be received via direct cabling (e.g., a local simulationparticipant's camera output connected to the computing system), via alocal area network (e.g., a local simulation participant camera outputdirected to a computing device and with the computing device connectedto the computing system over a local area network), via a wirelessnetwork (e.g., a local simulation participant camera output to awireless router associated with the computing system), and/or via a datanetwork such as the Internet (e.g., a geographically remote simulationparticipant). Optionally, motion tracking camera video output may bedirected to or received by a computing device (e.g., a smartphone, anetworked server, etc.) local to the simulation participant but whichmay be remote from the computing system. Optionally, the local computingdevice determines in whole or in part, for example using trackingsoftware, head location/position coordinates and location/positionchanges and only transmits head tracking location/position coordinatedata to the computing system (e.g., via access to shared networkedstorage, sent via a data network, etc.) and not the camera video output,thereby reducing the amount of data sent over a data network from aremote site and so reducing network bandwidth utilization. This approachmay reduce the need to have complex high performance networks.

For clarity, the description herein may refer to a user/operator whichmay receive enhanced tracking data, which is intended to encompass anyuser, operator, observer, participant, etc. which may be computingsystem configured to (or default automatically configured to) receiveenhanced tracking data, unless the context indicates otherwise. Inaddition, the descriptions herein may refer to the receipt of enhancedtracking data associated with a participant. Optionally, the enhancedtracking data may be associated with a plurality of participants. Inwhich case the system may associate respective identifiers with theplurality of participants, and, for example, may display the participantidentifier in association with a one or more conditions (e.g., aparticipant's eye's closing, a participant's veering from an element ofthe scene, etc.) on active displayed monitors. Optionally, in amulti-participant simulation, an alarm/indicator/notifier may betriggered and/or displayed to a user/operator when a configurablethreshold, such as a configurable number (e.g., a count of participants)or percentage of participant's in the simulation, trigger a condition(e.g., a count of 2 out of 4 participants in a presentation are notdirecting their gaze at the virtual slide screen or thepresenter/operator/user). As further described herein, a participant'sphysiology and/or actions may be data mined and/or stored forsubstantially real-time analysis and/or post analysis. Optionally, thedata is stored with participant identifiers.

Head Tracking

An optional aspect of an interactive, multi-user, virtual realitysession is the utilization of one or more motion tracking cameras thatrecord emitted infrared light pulses from tracking emitter devicesassociated with one or more virtual reality participants. An exampletracking device which emits trackable infrared light signatures is awireless Precision Position Tracking (PPT) device, PPT Eyes™, fromWorldViz LLC of Santa Barbara. PPT Eyes is a motion tracker mountable to3D glasses or virtual reality head mounted displays which provideswide-area tracking and can be integrated with or attached to displayheadgear enabling the tracking of a user's head movements when combinedwith virtual reality tracking software and one or motion trackingcameras, although other tracking devices may be used. Local and/orremote participants in a virtual reality simulation session may beassociated with passive and/or active head tracking components/devicesand one or motion tracking cameras that track the user (and the user'shead motions) and/or head tracking components/devices. Datacorresponding to tracked head movements is optionally used by thevirtual reality tracking software to cause scene renderings in aviewpoint of the user, and for displaying the movement of one or moreother participants in the simulation in corresponding viewed avatars.Advantageously, an aspect of the computing system optionally enables anoperator or a configured user to receive additional or enhanced headtracking data in substantially real-time (e.g., monitored head positionrelative to a target, head position changes as viewed from a positiondirectly above the participant, etc.) including head position changes(e.g., small changes in head positions, or changes in head position thatmay not be easily viewable from certain viewpoints) which may not beeasily discernable by a configured user and/or operator viewing anavatar representation of a participant. Optionally, motion trackingcamera video output may be received by the computing system directly orvia various wired or wireless networks or peripheral interfaces/buses.Optionally, motion tracking camera video output may be received viadirect cabling (e.g., a local simulation participant's camera outputconnected to the computing system), via a local area network (e.g., alocal simulation participant camera output directed to a computingdevice and with the computing device connected to the computing systemover a local area network), via a wireless network (e.g., a localsimulation participant camera output to a wireless router associatedwith the computing system), via a data network such as the Internet(e.g., a geographically remote simulation participant). Optionally,motion tracking camera video output may be directed to or received by acomputing device (e.g., a smartphone, a networked server, etc.) local tothe simulation participant but which may be remote from the computingsystem. Optionally, the local computing device determines in whole or inpart, for example using tracking software, head location/positioncoordinates and location/position changes and only transmits headtracking location/position coordinate data to the computing system(e.g., via access to shared networked storage, sent via a data network,etc.) and not the camera video output, thereby reducing the amount ofdata sent over a data network from a remote site and so reducing networkbandwidth utilization. This approach may reduce the need to have complexhigh performance networks. A further aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a visual head position dashboard wherein the dashboard isdisplayable in a portion of an operator's/user's display (e.g., headmounted display). Optionally, the dashboard displays aspects of theparticipant's head position, the participant's viewpoint, and/or certaininferences based upon an analysis of the head position data by itselfand/or in association with aspects of the simulation and/or receipt ofother enhanced monitoring/tracking data. A further aspect of thecomputing system enables an operator or a configured user to select oneor more participants, via a user interface control, from which enhancedhead tracking data may be received. Optionally, the system automaticallyprovides enhanced head tracking data to all users/operators/observers,and a user interface control is optionally provided to disable thepresentation of the tracking data. Optionally, the head trackingdashboard is visually presented in a portion of the screen inconjunction with audio cues or exclusively via audio cues (e.g., usingan audio alert tone indicating a monitored head position is no longerdirected at a target). Optionally, received head tracking position datais first data analyzed and/or compared against stored position data orhead tracking cues/signals in order to generate certain inferences asfurther described below before being made available to the operator orconfigured user. Optionally, received position data (including movementas determined from position data over a period of time) and or headtracking cues/signals are analyzed and/or compared by accessing internalor external data stores. Optionally, inferences are derived from acomparison of stored head position state changes and/or associated statechange patterns from stored historical simulations, generalized headtracking analysis (including non-virtual reality simulationenvironments), and/or other heuristic head tracking research results. Afurther optional aspect of the system enables theoperator/user/developer to define certain targets of interest in asimulation via a set of user interface controls which may condition aset of signals associated with a tracked head position of a participant.

With respect to identifying participants in which enhanced head trackingmay be received, optionally, a user interface control may be presentedon a display of an operator/user, for example in response to a controlselection by the user/operator. Optionally, in an enhanced head trackingselection mode state, all the participants in a simulation may bepresented (e.g., by name, alias, and/or avatar) on the display and auser/operator may select a participant presented on the user/operatordisplay. For example, the user/operator may select a participant via amouse-like wand control, a head movement directed at a target uservisible on the user/operator display (e.g., by a user/operator centeringa session participant in a target on their screen and then selecting acontrol), a pull-down list of participants, by speaking theparticipant's name, using voice command(s), or other user interfacecontrol selection. Optionally, the computing system automaticallydefaults enhanced head tracking to an operator/administrator (e.g. auser initiating the simulation, a server user/administrator in aclient-server configuration, a local user (versus a remote user), etc.).

With respect to the presentation of head tracking data in a displayformat, optionally, a dashboard or window display is projected on aportion of user/operator monitor (e.g., a head mounted display, computermonitor, etc.) of the user/operator which may displays certain headtracking data. Optionally, the user/operator may select a control toexpand or shrink the size of the dashboard display. Optionally, the sizeof the dashboard display may increase or decrease in response to certainconditions detected by the computing system. Optionally, the displaydashboard/window size (e.g., in the vertical and horizontal dimensions,or just in the vertical dimension, or just in the horizontal dimension)may be automatically increased at least partly in response to adetection of an increase in a subject's head movement relative to thesubject's mean head movement activity. Optionally, the displaydashboard/window size may be automatically increased at least partly inresponse to a detection that a participant's line of sight veers from oroff a designated target of interest. Optionally, the display window sizemay be increased at least partly in response to detection of certainstate changes of a designated target (e.g., an action of or associatedwith a designated target). Optionally, the display window size may beincreased or decreased at least partly in response to the occurrence ofcertain timed events or lack of head tracking activity. Optionally, atleast partly in response to a combination of one or more conditionsand/or one or more timed events, the display window size may beincreased or decreased. For example, the detection of a subject's headveering from a designated target condition for a fixed period of timemay cause the display window size to be increased. Certain state changesof the simulation itself may be visually taxing or may require moreattention of the user/operator, and in response to detecting theforegoing, the computing system may decrease the window display size(e.g., automatically). Optionally, a user interface control may beprovided which enables the user to instruct that the display window isto be moved to different areas of the monitor. Optionally, the displaywindow may be minimized in response to a user interface controlselection, condition(s), and/or time period expiration. Optionally, headtracking audio cues may be presented to a user whether the dashboarddisplay is minimized or not. Optionally, the user interface controlsdescribed herein include but are not limited to one or more of thefollowing: a hand gesture (e.g., a hand/limb gesture detected by atracking device), a hand selection by the user/operator of a control(e.g., a touch screen control, a wireless wand control, a computingdevice keyboard, etc.), a voice utterance control (e.g., an audio signalsuch as a whistle, grunt, etc.), a spoken word control (e.g., using aspecific voice command such as “smaller” or “minimize”), a facialgesture (e.g., a wink or two consecutive winks, open mouth, etc.), etc.Optionally, audio/voice control commands are optionally muted inresponse to the user selecting or speaking a mute signal (by speakingthe word “mute”), thereby preventing other simulation participants fromhearing spoken commands from the user. Optionally, the computing systemis configured with a configurable audio communication delay enabling thesystem to detect a control shift keyword (e.g., “Mute”) before streamingspeech from a user/operator over a communication channel with othersimulation participants. Optionally, the system provides a whisper mode,wherein detection of whispered commands or spoken audio below aspecified or determined volume threshold causes the system to preventsuch whispered commands or spoken audio from being transmitted to otherparticipants of the simulation.

Optionally, head tracking information is presented substantially inreal-time to a user/operator in the form of a status indicator light.For example, if a simulation session participant is in a trainingsession from which enhanced head tracking data is being received, asmall red indicator light may be presented (e.g., in a portion of theuser/operator's display) to a user in response to detecting that thesubject's (e.g. simulation participant's) head is no longer directed ata designated target of interest. Optionally, visual indicator cues maybe provided by the computing system only in response to detectingcertain specified target configuration variances and/or in the absenceof certain target configuration variances. For example, in the use caseabove, the computing system may display a continuous visual cue, such asa green light indicator, in response to detecting that the participant'shead is directed at a configured target of interest, and a red indicatormay be displayed in response to detecting that the participant's head isdirected elsewhere.

With respect to head tracking audio cues, optionally, an audio alert maybe generated in response to a head tracking state change. For example,in reference to the use case above, the system may generate a varianceaudio tone or whisper when a participant's head is directed away from aconfigured target. Optionally, a periodic (e.g., every 10 sec, 20 sec,30 sec, 1 min, etc.) compliance audio cue/signal is played to theuser/operator as long as the subject's head is directed at a configuredtarget.

As described herein, the computing system optionally monitors head statechanges of a participant and subject to certain configuredconstraints/limitations the user/operator is alerted. For example, auser/operator may be alerted in substantially real-time in response todetermining that the participant's head position and/or action is nolonger in an expected state. For example, if it detected that aparticipant's head begins to nod or nod repeatedly (as in nod off)during a presentation, the user/operator may be alerted to suchactivity. Optionally, head action is continually compared against aknowledge base which may enable the computing system to infer certaincharacteristics/attributes of the participant's emotional state orstate-of-mind. For example, in a training use case, detected up and downnods during a presentation may infer the session participant is inagreement with points being made by the presenter. Alternatively,detected side-to-side movements may indicate a session participant is indisagreement with aspects of the presentation. Optionally, head actiontracking is recorded during an entire simulation session. Optionally,head action tracking is only recorded during certain visual and/or audioevents and/or conditions (e.g., in response to a state change in thesimulation, an introduction of an element into the simulation, inresponse to a participant's head position change of state, etc.)

Optionally, enhanced head tracking data is recorded, stored, and datamined/analyzed subsequent to the simulation session. Optionally,participants are interviewed subsequent to the simulation session todetermine state-of-mind, reactions, attitudes, physiological, and otherpsychological criteria to create a knowledge base of head actions indifferent contextual virtual reality simulations. Optionally, the headactions and/or head action attributes (e.g., yes/no nods and/or count ofyes/no nods, period of time head is directed to an object of thesimulation, period of time a participant's head is direct towards theuser/operator, head leanings forward or back, rapid head actions (e.g.,in response to a simulation scene), etc.) may be integrated into a datastore to create a knowledge base of head actions corresponding to asimulation or other simulations. This knowledge base may be used to makecertain direct inferences from head tracking data or indirect inferenceswhen head tracking data is combined with other aspects of the simulation(e.g., a likelihood of a product or service sale, expected social mediafeedback, etc.) Optionally, a created knowledge base from virtualreality enhanced head tracking and/or general research related to headtracking may be used to score a session wherein the score is associatedwith the likelihood the session participant may take a certain action(e.g., purchase a good or service). Optionally, the computing systemrecords/logs related information associated with or at a time of aparticipant's head action. In association with a detected participant'shead action, the system may record some or all of the followinginformation: the time and date, a continuous voice snippet/phrase spokenprior to or during the head movement by the participant or otheruser/participant in the simulation, a recorded frame of the target,other sensory detected attributes associated with the participant (e.g.,heart rate, body temperature, perspiration, etc. as further describedbelow), other body and limb action and/or muscular reactions, etc.Optionally, the computing system converts received/recorded speech intotext enabling improved data mining including search and compare.Optionally, the computing system stores in computer memory recordedspeech and/or converted speech into text in association with auser/operator or participant identifier (e.g., a user name, accountnumber, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detected headaction is determined by the system to veer from or veer to a designatedtarget of interest. An optional aspect of the computing system enables auser/operator to designate an element of the simulation (includingduring the session) as a target of interest in which monitored feedbackmay be recorded and/or provided to the user/operator in substantiallyreal-time. Optionally, a simulation is developed with specific targetelements/objects in which tracking may be associated (e.g.,automatically). Optionally, the user/operator is presented with a listof target elements/objects prior to or at the launch of the simulationand may select a target via a user interface control (e.g., a pull downmenu of likely target elements/objects). Optionally, said target mayinclude a real-world object in an augmented reality scenario.Optionally, a participant and/or user/operator may be chosen as atarget. Optionally, the user/operator may remove a target (selected,preselected and/or predefined target) via a user interface control ofthe computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine an initial head orientation (e.g., a user head facing forward)from which the user's/participant's/subject's head position is trackedthroughout the simulation session.

Eye Tracking

An optional aspect of an interactive, multi-user, virtual realitysession is the utilization of eye tracking data of one or moreparticipants that may be collected via the utilization of one or morecameras or an eye tracking device, the latter which may provide a higherdegree of precision then a camera or one or more cameras. Optionally, aneye tracking device may include a light source (conventionally infrared)and a camera. The light source may be directed towards the eye(s) of theparticipant and the camera may record the reflection of the light sourcealong with visible ocular features such as the pupil. A computing devicein association with the camera or eye tracking device may determine thedirection of gaze or the eye position and may store the gaze and/or eyeposition or data corresponding to eye tracked movements and/or positionto computer memory. Advantageously, an aspect of the computing systemoptionally enables an operator or a configured user to receive eyetracking data in substantially real-time (e.g., monitored eye positionrelative to a target) including eye position changes (e.g., smallchanges in eye position) which may not be presented in association withan avatar or easily discernable by a configured user and/or operatorviewing an avatar representation of a participant. Optionally, the oneor more cameras and/or eye tracking device is integrated into or affixedto a head mounted display of the user/participant. Optionally, motiontracking camera video output/eye tracking device output may be receivedby the computing system via direct cabling (e.g., a local simulationparticipant's camera or eye tracking device output connected to thecomputing system), via a local area network (e.g., a local simulationparticipant camera output directed to a local computing device and withthe computing device connected to the computing system over a local areanetwork), via a wireless network (e.g., a local simulation participantcamera output to a wireless router associated with the computingsystem), via a data network including the Internet (e.g., a remotesimulation participant). Optionally, for example, the camera videooutput and/or eye tracking device output may be directed to a computingdevice (e.g., a smartphone, a networked server, etc.) local to thesimulation participant but which may be remote from the computingsystem. Optionally, the local computing device determines from thecaptured video or eye tracking output in whole or in part, for exampleusing tracking software, eye/gaze position data and only transmits saideye/gaze position data to the computing system (e.g., via access toshared networked storage, sent via a data network, etc.) therebyreducing the amount of data sent over the data network from the remotesite and so reducing network bandwidth utilization. This approach mayreduce the need to have complex high performance networks. For clarity,the description herein may refer to an eye, which is intended toencompass an entire eye or one or more elements of the eye (e.g.,pupil), unless the context indicates otherwise.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receive eyetracking data in substantially real-time which may not be providedand/or easily discernable by the operator/user viewing an avatarrepresentation of a participant. A aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a visual eye/gaze tracking dashboard displayable in a portionof an operator's/user's display (e.g., head mounted display).Optionally, the dashboard displays aspects of the participant's eyeposition and/or gaze, the participant's eye position data, and/orcertain inferences based upon an analysis of the eye position and/or eyemovement data in conjunction with aspects of the simulation and/orreceipt of other enhanced monitoring data. A further aspect of thecomputing system enables an operator or a configured user to select aparticipant, via a user interface control, in which eye/gaze positiondata may be received or only received. Optionally, the systemautomatically provides eye tracking data to allusers/operators/observers, and a user interface control may beoptionally provided whereby each user may disable the presentation ofthe enhanced tracking data. Optionally, the eye tracking dashboard isvisually presented in a portion of the screen to the user/operator withaudio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant may have fallen asleep). Optionally, receivedeye tracking position data is first data analyzed and/or comparedagainst stored position data or eye tracking cues/signals in order togenerating certain inferences as further described below before beingmade available to the operator or configured user. Optionally, receivedeye/gaze data analysis involves accessing historical eye/gaze dataand/or eye/gaze pattern data from one or more internal or external datastores including over wide-area data networks such as the Internet.Optionally, inferences are derived from a comparison of stored eye/gazeposition states and/or eye/gaze state changes/patterns from storedhistorical simulations, generalized eye/gaze tracking analysis(including non-virtual reality simulation environments), and/or otherheuristic eye/gaze tracking research results. A further optional aspectof the system enables the operator and/or user via a set of userinterface controls to configure certain eye tracking target options.

With respect to identifying participants in which enhanced eye/gazetracking may be received, optionally, a user interface control may bepresented on a display of an operator/user, for example in response to acontrol selection by the user/operator. Optionally, in an enhancedeye/gaze tracking selection mode state, all the participants in asimulation may be presented (e.g., by name, alias, and/or avatar) on thedisplay and a user/operator may select a participant presented on theuser/operator display. For example, the user/operator may select aparticipant via a mouse-like wand control, a head movement directed at atarget user visible on the user/operator display (e.g., by auser/operator centering a session participant in a target on theirscreen and then selecting a control), a pull-down list of participants,by speaking the participant's name, using voice command(s), or otheruser interface control selection. Optionally, the computing systemautomatically defaults enhanced eye/gaze tracking to anoperator/administrator (e.g. a user initiating the simulation, a serveruser/administrator in a client-server configuration, a local user(versus a remote user), etc.).

With respect to the presentation of eye/gaze tracking data in a displayformat, optionally, a dashboard or window display is projected on aportion of user/operator's display (e.g., a head mounted display,computer monitor, etc.) of the user/operator which may displays certaineye/gaze tracking data. Optionally, the user/operator may select acontrol to expand or shrink the size of the dashboard display.Optionally, the size of the dashboard display may increase or decreasein response to certain conditions detected by the computing system.Optionally, the display dashboard/window size (e.g., in the vertical andhorizontal dimensions, or just in the vertical dimension, or just in thehorizontal dimension) may be automatically increased at least partly inresponse to a detection of an increase in a subject's eye/gaze movementrelative to the subject's mean eye/gaze movement activity. Optionally,the display dashboard/window size may be automatically increased atleast partly in response to a detection that a participant's line ofsight veers from or off a designated target(s) of interest. Optionally,the display window size may be increased at least partly in response todetection of certain state changes of a designated target (e.g., anaction of or associated with a designated target). Optionally, thedisplay window size may be increased or decreased at least partly inresponse to the occurrence of certain timed events or lack of eye/gazetracking activity. Optionally, at least partly in response to acombination of one or more conditions and/or one or more timed events,the display window size may be increased or decreased. For example, thedetection of a subject's eye/gaze veering from a designated targetcondition for a fixed period of time may cause the display window sizeto be increased. Certain state changes of the simulation itself may bevisually taxing or may require more attention of the user/operator, andin response to detecting the foregoing, the computing system maydecrease the window display size (e.g., automatically). Optionally, auser interface control may be provided which enables the user toinstruct that the display window is to be moved to different areas ofthe monitor. Optionally, the display window may be minimized in responseto a user interface control selection, condition(s), and/or time periodexpiration. Optionally, eye/gaze tracking audio cues may be presented toa user whether the dashboard display is minimized or not. Optionally,the user interface controls described herein include but are not limitedto one or more of the following: a hand gesture (e.g., a hand/limbgesture detected by a tracking device), a hand selection by theuser/operator of a control (e.g., a touch screen control, a wirelesswand control, a computing device keyboard, etc.), a voice utterancecontrol (e.g., an audio signal such as a whistle, grunt, etc.), a spokenword control (e.g., using a specific voice command such as “smaller” or“minimize”), a facial gesture (e.g., a wink or two consecutive winks,open mouth, etc.), etc. Optionally, audio/voice control commands areoptionally muted in response to the user selecting or speaking a mutesignal (by speaking the word “mute”), thereby preventing othersimulation participants from hearing spoken commands from the user.Optionally, the computing system is configured with a configurable audiocommunication delay enabling the system to detect a control shiftkeyword (e.g., “Mute”) before streaming speech from a user/operator overa communication channel with other simulation participants. Optionally,the system provides a whisper mode, wherein detection of whisperedcommands or spoken audio below a specified or determined volumethreshold causes the system to prevent such whispered commands or spokenaudio from being transmitted to other participants of the simulation.

Optionally, eye/gaze tracking information is presented substantially inreal-time to a user/operator in the form of a status indicator light.For example, if a simulation session participant(s) is in a trainingsession from which enhanced eye/gaze tracking data is being received, asmall red indicator light may be presented (e.g., in a portion of theuser/operator's display) to a user in response to detecting that thesubject's (e.g. simulation participant's) eye/gaze is no longer directedat a designated target of interest. Optionally, visual indicator cuesmay be provided by the computing system only in response to detectingcertain specified target configuration variances and/or in the absenceof certain target configuration variances. For example, in the use caseabove, the computing system may display a continuous visual cue, such asa green light indicator, in response to detecting that the participant'seye/gaze is directed at a configured target of interest, and a redindicator may be displayed in response to detecting that theparticipant's eye/gaze is directed elsewhere.

With respect to eye/gaze tracking audio cues, optionally, an audio alertmay be generated in response to an eye/gaze tracking state change. Forexample, in reference to the use case above, the system may generate avariance audio tone or whisper when a participant's eye/gaze is directedaway from a configured target. Optionally, a periodic (e.g., every 10sec, 20 sec, 30 sec, 1 min, etc.) compliance audio cue/signal is playedto the user/operator as long as the subject's eye/gaze is directed at aconfigured target.

As described herein, the computing system optionally monitors eye/gazestate changes of a participant and subject to certain configuredconstraints/limitations the user/operator is alerted. For example, auser/operator may be alerted in substantially real-time in response todetermining that the participant's eye/gaze position and/or action is nolonger in an expected state. For example, if it detected that aparticipant eye's are partially closed or closed during a presentation,the user/operator may be alerted to such activity. Optionally, eye/gazeaction is continually compared against a knowledge base which may enablethe computing system to infer certain characteristics/attributes of theparticipant's emotional state or state-of-mind. For example, in atraining use case, an increased eye opening during a presentation mayinfer the session participant is in agreement with points being made bythe presenter. Alternatively, detected eye squinting may indicate asession participant is in disbelief with aspects of the presentation.Optionally, eye/gaze action tracking is recorded during an entiresimulation session. Optionally, eye/gaze action tracking is onlyrecorded during certain visual and/or audio events and/or conditions(e.g., in response to a state change in the simulation, an introductionof an element into the simulation, in response to a participant'seye/gaze position change of state, etc.)

Optionally, enhanced eye/gaze tracking data is recorded, stored, anddata mined/analyzed subsequent to the simulation session. Optionally,participants are interviewed subsequent to the simulation session todetermine state-of-mind, reactions, attitudes, physiological, and otherpsychological criteria to create a knowledge base of eye/gaze actions indifferent contextual virtual reality simulations. Optionally, theeye/gaze actions and/or eye/gaze action attributes (e.g., eye lidclosure and/or count of eye lid closures, period of time eye/gaze isdirected to an object of the simulation, period of time a participant'seye/gaze is direct towards the user/operator, number and rate of blinks,rapid eye/gaze actions (e.g., in response to a simulation scene), etc.)may be integrated into a data store to create a knowledge base ofeye/gaze actions corresponding to a simulation or other simulations.This knowledge base may be used to make certain direct inferences fromeye/gaze tracking data or indirect inferences when eye/gaze trackingdata is combined with other aspects of the simulation (e.g., alikelihood of a product or service sale, expected social media feedback,etc.) Optionally, a created knowledge base from virtual reality enhancedeye/gaze tracking and/or general research related to eye/gaze trackingmay be used to score a session wherein the score is associated with thelikelihood the session participant may take a certain action (e.g.,purchase a good or service). Optionally, the computing systemrecords/logs related information associated with or at a time of aparticipant's eye/gaze action. In association with a detectedparticipant's eye/gaze action, the system may record some or all of thefollowing information: the time and date, a continuous voicesnippet/phrase spoken prior to or during the eye/gaze movement by theparticipant or other user/participant in the simulation, a recordedframe of the target, other sensory detected attributes associated withthe participant (e.g., heart rate, body temperature, perspiration, etc.as further described below), other body and limb action and/or muscularreactions, etc. Optionally, the computing system convertsreceived/recorded speech into text enabling improved data miningincluding search and compare. Optionally, the computing system stores incomputer memory recorded speech and/or converted speech into text inassociation with a user/operator or participant identifier (e.g., a username, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detectedeye/gaze action is determined by the system to veer from or veer to adesignated target of interest. An optional aspect of the computingsystem enables a user/operator to designate an element of the simulation(including during the session) as a target of interest in whichmonitored feedback may be recorded and/or provided to the user/operatorin substantially real-time. Optionally, a simulation is developed withspecific target elements/objects in which tracking may be associated(e.g., automatically). Optionally, the user/operator is presented with alist of target elements/objects prior to or at the launch of thesimulation and may select a target via a user interface control (e.g., apull down menu of likely target elements/objects). Optionally, saidtarget may include a real-world object in an augmented reality scenario.Optionally, a participant and/or user/operator may be chosen as atarget. Optionally, the user/operator may remove a target (selected,preselected and/or predefined target) via a user interface control ofthe computing system.

As previously described herein, the presentation of eye/gaze trackingdata is projected on a portion of user/operator's display (e.g., a headmounted display, computer monitor, etc.) of the user/operator.Optionally, for example to give the user/operator visibility into thedirectional gaze of the participant, the presentation is an imagedisplay of a designated participant's eyes as recorded by a cameraassociated with the participant. Optionally, the displayed presentationis a representation of a designated participant's eyes as illustrated inFIG. 3. Optionally, if there are two or more participant eye displays, aparticipant identifier is displayed in association with the eyedisplays. Optionally, a display of participant's eyes is directed at theeyes of the user/operator when the participant's gaze is determined tobe directed at the user/operator or a target element of the scene of thesimulation. Optionally, an initial calibration step is performed toestablish participant eye alignment with a user/operator and/or targetelement of the scene.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine an initial eye/gaze orientation (e.g., a user eye/gaze facingforward) from which the user's/participant's/subject's eye/gaze positionis tracked throughout the simulation session.

Facial Expressions

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of facial expression tracking data of oneor more participant that may be collected via the utilization of one ormore cameras, or other facial expression capture devices. Facialexpressions communicate certain nonverbal aspects of auser/participant's state of mind and/or emotional state. Conventionally,emotional states fall into six states of angry, happy, sad, fearful,surprised, and disgusted. However, recent research into facialexpression analysis has tied certain consistent facial muscle movementinto additional emotional states including, for example, “happilydisgusted” or “sadly angry.” Advantageously, the computing system mayderive certain inferences from a facial expression state and/or facialexpression state changes (e.g., reactions to certain simulation events)which may be provided to a user/operator/observer during and subsequentto the virtual reality simulation. A computing device in associationwith the camera may capture facial expressions of the virtual realityparticipant including any changes to a participant's facial expressionsand may store the actual recorded facial expressions to computer memoryor data corresponding to detected facial expressions. Advantageously, anaspect of the computing system optionally enables an operator or aconfigured user to receive facial expression data or an inferredemotional state in substantially real-time (e.g., a smile or thecorresponding indicative emotion “happy”) including facial expressionchanges (e.g., a neutral expression to a smile) which may not bepresented in association with an avatar or easily discernable by aconfigured user and/or operator viewing an avatar representation of aparticipant. Optionally, the one or more cameras are integrated into oraffixed to a head mounted display of the user/participant. Optionally,the facial expression camera output may be received by the computingsystem via direct cabling (e.g., a local simulation participant's cameraoutput connected to the computing system), via a local area network(e.g., a local simulation participant camera output directed to a localcomputing device connected to the computing system over a local areanetwork), via a local wireless network (e.g., a local simulationparticipant camera output to a wireless router associated with thecomputing system), via a data network including the Internet (e.g., aremote simulation participant). Optionally, for example, the cameraoutput may be directed to a computing device (e.g., a smartphone, anetworked server, etc.) local to the simulation participant but whichmay be remote from the computing system. Optionally, the local computingdevice determines from the captured video, facial expression output inwhole or in part, for example using facial expression softwareanalytics, and only transmits said expression data analysis to thecomputing system (e.g., via access to shared networked storage, sent viaa data network, etc.) thereby reducing the amount of data sent over thedata network from the remote site and so reducing network bandwidthutilization. This approach may reduce the need to have complex highperformance networks. For clarity, the description herein may refer to aface or facial, which is intended to encompass an entire face or one ormore elements of the face (e.g., lips, mouth, nose, muscles under skinof the face, nose, forehead, eyebrows, eyes, ears, etc.), unless thecontext indicates otherwise.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receivefacial expression data in substantially real-time which may not beprovided and/or easily discernable by the operator/user viewing anavatar representation of a participant. An aspect of the computingsystem enables an operator or a configured user to be presented with,for example, a visual facial expression state dashboard displayable in aportion of an operator's/user's display (e.g., head mounted display).Optionally, the dashboard displays aspects of the participant's facialexpressions, the participant's facial expression data, and/or certaininferences based upon an analysis of the participant's face data inconjunction with aspects of the simulation and/or receipt of otherenhanced monitoring data. Optionally, the dashboard display includes animage display of a designated participant's face as recorded by a cameraassociated with the participant. Optionally, the displayed presentationis a representation of a designated participant's face. Optionally, ifthere are two or more participant facial displays, a participantidentifier is displayed in association with the facial displays. Anexample of a facial expression inference is inferring that a participantis pleased in response to the computing system determining that theparticipant is smiling. A further aspect of the computing system enablesan operator or a configured user to select a participant, via a userinterface control, in which facial expression data may be received.Optionally, the system automatically provides facial expression data toall users/operators/observers, and a user interface control may beoptionally provided whereby each user may disable the presentation ofthe enhanced tracking data. Optionally, the facial expression dashboardis visually presented in a portion of the screen to the user/operatorwith audio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant may have fallen asleep). Optionally, receivedfacial expression position data is first data analyzed and/comparedagainst stored position data or eye tracking cues/signals in order togenerating certain inferences as further described below before beingmade available to the operator or configured user. Optionally, thereceived expression data analysis involves accessing historical data orexpression template data from one or more internal or external datastores including over wide-area data networks such as the Internet.Optionally, inferences are derived from a comparison of stored facialexpressions and/or facial expression state changes and/or associatedstate change patterns from stored historical simulations, generalizedfacial expression tracking analysis (including non-virtual realitysimulation environments), and/or other heuristic facial expressiontracking research results. A further optional aspect of the systemenables the operator and/or user via a set of user interface controls todetect certain facial expression state options.

With respect to identifying participants in which enhanced facialexpression monitoring may be received, optionally, a user interfacecontrol may be presented on a display of an operator/user, for examplein response to a control selection by the user/operator. Optionally, inan enhanced facial expression monitoring selection mode state, all theparticipants in a simulation may be presented (e.g., by name, alias,and/or avatar) on the display and a user/operator may select aparticipant presented on the user/operator display. For example, theuser/operator may select a participant via a mouse-like wand control, ahead movement directed at a target user visible on the user/operatordisplay (e.g., by a user/operator centering a session participant in atarget on their screen and then selecting a control), a pull-down listof participants, by speaking the participant's name, using voicecommand(s), or other user interface control selection. Optionally, thecomputing system automatically defaults enhanced facial expressionmonitoring to an operator/administrator (e.g. a user initiating thesimulation, a server user/administrator in a client-serverconfiguration, a local user (versus a remote user), etc.).

With respect to the presentation of facial expression monitoring data ina display format, optionally, a dashboard or window display is projectedon a portion of user/operator display (e.g., a head mounted display,computer monitor, etc.) of the user/operator which may displays certainfacial expression monitoring data. Optionally, the user/operator mayselect a control to expand or shrink the size of the dashboard display.Optionally, the size of the dashboard display may increase or decreasein response to certain conditions detected by the computing system.Optionally, the display dashboard/window size (e.g., in the vertical andhorizontal dimensions, or just in the vertical dimension, or just in thehorizontal dimension) may be automatically increased at least partly inresponse to a determination of a certain type of expression which mayindicate, for example, a certain emotional state. Optionally, thedisplay window size may be increased at least partly in response todetection of certain expression state changes corresponding to an actionof a target. Optionally, the display window size may be increased ordecreased at least partly in response to the occurrence of certain timedevents or lack of certain facial expression monitoring activity.Optionally, at least partly in response to a combination of one or moreconditions and/or one or more timed events, the display window size maybe increased or decreased. For example, the detection of a subject'sexpression not changing for a fixed period of time may cause the displaywindow size to be increased. Certain state changes of the simulationitself may be visually taxing or may require more attention of theuser/operator, and in response to detecting the foregoing, the computingsystem may decrease the window display size (e.g., automatically).Optionally, the dashboard may display a real-time view of theparticipant's face. Optionally, the dashboard may only display areal-time view of the participant's face in response to a condition,such as an expression state change. Optionally, a user interface controlmay be provided which enables the user to configure the conditions(e.g., change of inferred emotion change, happy, sad, angry, confused,etc.) when a view of the participant's face may be displayed and for howlong. Optionally, a user interface control may be provided which enablesthe user to instruct that the display window is to be moved to differentareas of the monitor. Optionally, the display window may be minimized inresponse to a user interface control selection, condition(s), and/ortime period expiration. Optionally, facial expression monitoring audiocues may be presented to a user whether the dashboard display isminimized or not. Optionally, the user interface controls describedherein include but are not limited to one or more of the following: ahand gesture (e.g., a hand/limb gesture detected by a tracking device),a hand selection by the user/operator of a control (e.g., a touch screencontrol, a wireless wand control, a computing device keyboard, etc.), avoice utterance control (e.g., an audio signal such as a whistle, grunt,etc.), a spoken word control (e.g., using a specific voice command suchas “smaller” or “minimize”), a facial gesture (e.g., a wink or twoconsecutive winks, open mouth, etc.), etc. Optionally, audio/voicecontrol commands are optionally muted in response to the user selectingor speaking a mute signal (by speaking the word “mute”), therebypreventing other simulation participants from hearing spoken commandsfrom the user. Optionally, the computing system is configured with aconfigurable audio communication delay enabling the system to detect acontrol shift keyword (e.g., “Mute”) before streaming speech from auser/operator over a communication channel with other simulationparticipants. Optionally, the system provides a whisper mode, whereindetection of whispered commands or spoken audio below a specified ordetermined volume threshold causes the system to prevent such whisperedcommands or spoken audio from being transmitted to other participants ofthe simulation.

Optionally, facial expression monitoring information is presentedsubstantially in real-time to a user/operator in the form of a statusindicator light. For example, if a simulation session participant is ina training session from which enhanced facial expression monitoring datais being received, a small red indicator light may be presented (e.g.,in a portion of the user/operator's display) to a user in response tothe computing system determining that the subject's (e.g. simulationparticipant's) expression indicates a certain emotional state, forexample, anger or disappointment. Optionally, visual indicator cues maybe provided by the computing system only in response to detectingcertain specified expression variances and/or in the absence of certainexpression variances. For example, in the use case above, the computingsystem may display a continuous visual cue, such as a green lightindicator, in response to determining that a participant's expressionmay indicate the participant may be in a relaxed and/or happy state, anda red indicator may be displayed in response to determining that theparticipant's expression may be indicative of a tense or angry state.

With respect to facial expression monitoring audio cues, optionally, anaudio alert may be generated in response to a facial expressionmonitoring state change. For example, in reference to the use caseabove, the system may generate a variance audio tone or whisper when thecomputing system determines a participant's expression may have changedfrom an indication of a participant's relaxed state to a tense or angrystate. Optionally, a periodic (e.g., every 10 sec, 20 sec, 30 sec, 1min, etc.) compliance audio cue/signal is played to the user/operator aslong as the subject's expression may indicate, for example, a relaxed orhappy state.

As described herein, the computing system optionally monitors expressionstate changes of a participant and subject to certain configuredconstraints/limitations the user/operator is alerted. For example, auser/operator may be alerted in substantially real-time in response todetermining that the participant's facial expression is no longer in anexpected state. For example, if it is determined that a participantexpressions are indicative of an unhappy or disappointed state during apresentation, the user/operator may be alerted to such state of theparticipant. Optionally, expression data is continually compared againsta knowledge base which may enable the computing system to infer certaincharacteristics/attributes of the participant's emotional state orstate-of-mind. For example, in a training use case, a smile during apresentation may infer the session participant is in agreement withpoints being made by the presenter or liked a joke. Alternatively,detected pursed lips may indicate a session participant is in disbeliefwith aspects of the presentation. Optionally, facial expressionmonitoring may be recorded during an entire simulation session.Optionally, facial expression monitoring may only be recorded duringcertain visual and/or audio events and/or conditions (e.g., in responseto a state change in the simulation, an introduction of an element intothe simulation, in response to a participant's expression change ofstate, etc.)

Optionally, enhanced facial expression monitoring data is recorded,stored, and data mined/analyzed subsequent to the simulation session.Optionally, participants are interviewed subsequent to the simulationsession to determine state-of-mind, reactions, attitudes, physiological,and other psychological criteria to create a knowledge base ofexpressions in different contextual virtual reality simulations.Optionally, the expressions actions and/or expressions attributes (e.g.,lip movements, cheek movements, eye brow movements, etc.) may beintegrated into a data store to create a knowledge base of expressionand/or expression attributes corresponding to a participant'sstate-of-mind and/or emotion during a simulation or other simulations.This knowledge base may be used to make certain direct inferences fromfacial expression monitoring data or indirect inferences when facialexpression monitoring data may be combined with other aspects of thesimulation (e.g., a likelihood of a product or service sale, expectedsocial media feedback, etc.) Optionally, a created knowledge base fromvirtual reality enhanced facial expression monitoring and/or generalresearch related to facial expression monitoring may be used to score asession wherein the score is associated with the likelihood the sessionparticipant may take a certain action (e.g., purchase a good orservice). Optionally, the computing system records/logs relatedinformation associated with or at a time of a participant's expressionchange of state. In association with a detected participant's expressionchange of state, the system may record some or all of the followinginformation: the time and date, a continuous voice snippet/phrase spokenprior to or during the expression by the participant or otheruser/participant in the simulation, other sensory detected attributesassociated with the participant (e.g., heart rate, body temperature,perspiration, etc. as further described below), other body and limbaction and/or muscular reactions, etc. Optionally, the computing systemconverts received/recorded speech into text enabling improved datamining including search and compare. Optionally, the computing systemstores in computer memory recorded speech and/or converted speech intotext in association with a user/operator or participant identifier(e.g., a user name, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detectedexpression (or emotional state) is determined by the system to changefrom a configured state. An optional aspect of the computing systemenables a user/operator to designate an emotional state of a participantin which monitored feedback may be recorded and/or provided to theuser/operator in substantially real-time. Optionally, a simulation isdeveloped with specific emotional state in which tracking may beassociated (e.g., automatically). Optionally, the user/operator ispresented with a list of emotional states prior to or at the launch ofthe simulation and may select an emotional state via a user interfacecontrol (e.g., a pull down menu of likely emotional states). Optionally,the user/operator may remove said selected options via a user interfacecontrol of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine facial orientation, facial attributes, facial position, etc.from which the user's/participant's/subject's facial expressionmonitoring is tracked throughout the simulation session.

Monitors—Heart

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of heart data of a participant collectedusing a heart sensor. The tracked heart data of a participant may beused by itself or in combination with other monitored factors (e.g.,facial expressions) to determine aspects of the participant including,for example, the emotional state of the participant. For example, adetected and/or determined increased heart rate may indicate anxiety,apprehension, excitement, etc. of a participant in a portion of thesimulation in which the increased heart rate was detected and/ordetermined. Advantageously, the computing system may derive certaininferences from a heart state and/or heart state changes (e.g.,reactions to certain simulation events) which may be displayed to auser/operator/observer during and/or subsequent to a virtual realitysimulation. A computing device in association with a heart sensor of thevirtual reality participant may record aspects of the participant'sheart activity and/or any changes and may store the actual sensorrecordings to computer memory or data corresponding to the participant'ssensor output. Advantageously, an aspect of the computing systemoptionally enables an operator or a configured user to receive heartdata or inferences corresponding to the heart data in substantiallyreal-time (e.g., the likelihood of an answer being truthful) includingheart changes (e.g., an increase in a participant's heart rate) whichmay not be presented in association with an avatar or easily discernableby a configured user and/or operator viewing an avatar representation ofa participant. Optionally, the heart sensor may be integrated into oraffixed to a head mounted display of the user/participant. Optionally,the heart sensor may be worn in the form of a chest or wrist strap.Optionally, heart sensor data may be received by the computing systemwirelessly to a computing device of the user/participant (e.g., asmartwatch, smartphone, etc.) Optionally, heart sensor data may bereceived by the computing system via direct cabling (e.g., a localsimulation participant sensor output connected to the computing system),via a local area network (e.g., a local simulation participant sensoroutput directed to a local computing device (e.g., a computing device ofthe user) connected to the computing system over a local area network),via a local wireless network (e.g., a local simulation participantsensor output to a wireless router associated with the computingsystem), via a wide-area data network including the Internet (e.g.,receipt of sensor data from a remote participant's computing device (ora computing device of the computing system) wherein the sensor data iswirelessly transmitted to a local computing device of the participant).Optionally, the local computing device determines certain heartactivity-based inferences from the captured sensor data output in wholeor in part, for example using heart sensor software analytics, and onlytransmits said data inferences to the computing system (e.g., via accessto shared networked storage, sent via a data network, etc.) therebyreducing the amount of data sent over the data network from the remotesite and so reducing network bandwidth utilization. This approach mayreduce the need to have complex high performance networks. For clarity,the description herein may refer to a heart sensor, which is intended toencompass the detection/determination of a participant's heart activityincluding but not limited to a simple heart beat rate, blood pressure,electrical activity of the heart (conventionally referred to aselectrocardiography or EKG, ECG) etc., unless the context indicatesotherwise.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant heart data in substantially real-time which may not beprovided to the operator/user. An aspect of the computing system enablesan operator or a configured user to be presented with, for example, aheart state dashboard displayable in a portion of an operator's/user'sdisplay (e.g., head mounted display). Optionally, the dashboard displaysaspects of the participant's heart activity, and/or certain inferencesbased upon an analysis of the participant's heart activity data inconjunction with aspects of the simulation and/or receipt of otherenhanced monitoring data. Optionally, the dashboard display includes arepresentative display of a pumping heart. Optionally, if there are twoor more participant heart activity displays, a participant identifier isdisplayed in association with the facial displays. An example of a heartactivity inference is inferring that a participant is pleased inresponse to determining from the sensor that the heart rate is low. Afurther aspect of the computing system enables an operator or aconfigured user to select a participant, via a user interface control,in which heart data may be received. Optionally, the systemautomatically provides heart data to all users/operators/observers, anda user interface control may be optionally provided whereby each usermay disable the presentation of the enhanced tracking data. Optionally,the heart dashboard is visually presented in a portion of the screen tothe user/operator with audio cues or exclusively via audio cues (e.g.,using an audio tone indicating a participant's heart rate may haveincreased above a threshold or deviated from a recorded average of theparticipant). Optionally, received heart data is first data analyzedand/or compared against stored heart activity data or heart activitytracking cues/signals/patterns in order to generating certain inferencesas further described below before being made available to the operatoror configured user. Optionally, received heart data analysis involvesaccessing historical data and/or heart activity pattern data from one ormore internal or external data stores including over wide-area datanetworks such as the Internet. Optionally, inferences are derived from acomparison of stored heart state and/or heart state changes/patternsfrom stored historical simulations, generalized heart tracking analysis(including non-virtual reality simulation environments), and/or otherheuristic heart tracking research results. A further optional aspect ofthe system enable the operator and/or user via a set of user interfacecontrols to detect certain heart state target options or patterns.

With respect to identifying participants in which enhanced heartmonitoring may be received, optionally, a user interface control may bepresented on a display of an operator/user, for example in response to acontrol selection by the user/operator. Optionally, in an enhanced heartmonitoring selection mode state, all the participants in a simulationmay be presented (e.g., by name, alias, and/or avatar) on the displayand a user/operator may select a participant presented on theuser/operator display. For example, the user/operator may select aparticipant via a mouse-like wand control, a head movement directed at atarget user visible on the user/operator display (e.g., by auser/operator centering a session participant in a target on theirscreen and then selecting a control), a pull-down list of participants,by speaking the participant's name, using voice command(s), or otheruser interface control selection. Optionally, the computing systemautomatically defaults enhanced heart monitoring to anoperator/administrator (e.g. a user initiating the simulation, a serveruser/administrator in a client-server configuration, a local user(versus a remote user), etc.).

With respect to the presentation of heart monitoring data in a displayformat, optionally, a dashboard or window display is projected on aportion of user/operator display (e.g., a head mounted display, computermonitor, etc.) of the user/operator which may displays certain heartmonitoring data. Optionally, the user/operator may select a control toexpand or shrink the size of the dashboard display. Optionally, the sizeof the dashboard display may increase or decrease in response to certainconditions detected by the computing system. Optionally, the displaydashboard/window size (e.g., in the vertical and horizontal dimensions,or just in the vertical dimension, or just in the horizontal dimension)may be automatically increased at least partly in response to adetermination of a certain type of heart rate activity pattern which mayindicate, for example, a certain emotional state. Optionally, thedisplay window size may be increased at least partly in response todetection of certain heart activity state changes corresponding to anaction of a target. Optionally, the display window size may be increasedor decreased at least partly in response to the occurrence of certaintimed events or lack of certain heart monitoring activity. Optionally,at least partly in response to a combination of one or more conditionsand/or one or more timed events, the display window size may beincreased or decreased. For example, the detection of a subject's heartrate activity not changing for a fixed period of time may cause thedisplay window size to be increased. Certain state changes of thesimulation itself may be visually taxing or may require more attentionof the user/operator, and in response to detecting the foregoing, thecomputing system may decrease the window display size (e.g.,automatically). Optionally, the dashboard may display a real-time view,for example, of the participant's heart rate as a number and inassociation with a continuous graph of heart beats. Optionally, thedashboard may only display a real-time view of the participant's heartactivity in response to a condition, such as a heart activity statechange. Optionally, a user interface control may be provided whichenables the user to configure the conditions (e.g., heart rate greaterthan mean, heart rate less than mean, heart rate greater than athreshold or less than a threshold rate, etc.) when a view of theparticipant's heart activity may be displayed and for how long.Optionally, a user interface control may be provided which enables theuser to instruct that the display window is to be moved to differentareas of the monitor. Optionally, the display window may be minimized inresponse to a user interface control selection, condition(s), and/ortime period expiration. Optionally, heart monitoring audio cues may bepresented to a user whether the dashboard display is minimized or not.Optionally, the user interface controls described herein include but arenot limited to one or more of the following: a hand gesture (e.g., ahand/limb gesture detected by a tracking device), a hand selection bythe user/operator of a control (e.g., a touch screen control, a wirelesswand control, a computing device keyboard, etc.), a voice utterancecontrol (e.g., an audio signal such as a whistle, grunt, etc.), a spokenword control (e.g., using a specific voice command such as “smaller” or“minimize”), a facial gesture (e.g., a wink or two consecutive winks,open mouth, etc.), etc. Optionally, audio/voice control commands areoptionally muted in response to the user selecting or speaking a mutesignal (by speaking the word “mute”), thereby preventing othersimulation participants from hearing spoken commands from the user.Optionally, the computing system is configured with a configurable audiocommunication delay enabling the system to detect a control shiftkeyword (e.g., “Mute”) before streaming speech from a user/operator overa communication channel with other simulation participants. Optionally,the system provides a whisper mode, wherein detection of whisperedcommands or spoken audio below a specified or determined volumethreshold causes the system to prevent such whispered commands or spokenaudio from being transmitted to other participants of the simulation.

Optionally, heart monitoring information is presented substantially inreal-time to a user/operator in the form of a status indicator light.For example, if a simulation session participant is in a trainingsession from which enhanced heart monitoring data is being received, asmall red indicator light may be presented (e.g., in a portion of theuser/operator's display) to a user in response to the computing systemdetermining that the subject's (e.g. simulation participant's) heartactivity is above a certain range or indicates a certain emotionalstate, for example, anger or excitement. Optionally, visual indicatorcues may be provided by the computing system only in response todetecting certain specified heart activity variances and/or in theabsence of certain heart rate activity variances. For example, in theuse case above, the computing system may display a continuous visualcue, such as a green light indicator, in response to determining that aparticipant's heart activity may indicate the participant may be in arelaxed and/or happy state, and a red indicator may be displayed inresponse to determining that the participant's heart activity may beindicative of a tense or angry state.

With respect to heart monitoring audio cues, optionally, an audio alertmay be generated in response to a heart monitoring state change. Forexample, in reference to the use case above, the system may generate avariance audio tone or whisper when the computing system determines aparticipant's heart activity may have changed from an indication of aparticipant's relaxed state to a tense or angry state. Optionally, aperiodic (e.g., every 10 sec, 20 sec, 30 sec, 1 min, etc.) complianceaudio cue/signal is played to the user/operator as long as the subject'sheart activity may indicate, for example, a relaxed or happy state.

As described herein, the computing system optionally monitors heartactivity state changes of a participant and subject to certainconfigured constraints/limitations the user/operator is alerted. Forexample, a user/operator may be alerted in substantially real-time inresponse to determining that the participant's body temperature is nolonger in an expected state. For example, it may be determined that aparticipant heart activity are indicative of an unhappy or disappointedstate during a presentation, the user/operator may be alerted to suchstate of the participant. Optionally, heart activity may be continuallycompared against a knowledge base which may enable the computing systemto infer certain characteristics/attributes of the participant'semotional state or state-of-mind. For example, in a training use case, alow heart rate activity during a presentation may infer the sessionparticipant is relaxed and in general agreement with points being madeby the presenter or may be bored. Alternatively, a high activity heartrate may indicate a session participant is in disbelief with aspects ofthe presentation, excited and/or angry. As the foregoing two examplesillustrate, the heart activity may be combined with other monitoredactivity (e.g., facial expressions) to determine the participant'semotional state or state-of-mind (e.g., a high heart activity combinedwith pursed lips may indicate disbelief in certain aspects of apresentation). Optionally, heart monitoring may be recorded during anentire simulation session. Optionally, heart monitoring may only berecorded during certain visual and/or audio events and/or conditions(e.g., in response to a state change in the simulation, an introductionof an element into the simulation, in response to a change in aparticipant's heart activity, etc.)

Optionally, enhanced heart monitoring data is recorded, stored, and datamined/analyzed subsequent to the simulation session. Optionally,participants are interviewed subsequent to the simulation session todetermine state-of-mind, reactions, attitudes, physiological, and otherpsychological criteria to create a knowledge base of heart activity indifferent contextual virtual reality simulations. Optionally, the heartactivity patterns and/or heart activity attributes (e.g., heart rate,blood pressure readings, heart electrical activity, etc.) may beintegrated into a data store to create a knowledge base of heartactivity and/or heart activity attributes corresponding to aparticipant's state-of-mind and/or emotion during a simulation or othersimulations. This knowledge base may be used to make certain directinferences from heart monitoring data or indirect inferences when heartmonitoring data may be combined with other aspects of the simulation(e.g., a likelihood of a product or service sale, expected social mediafeedback, etc.) Optionally, a created knowledge base from virtualreality enhanced heart monitoring and/or general research related toheart monitoring may be used to score a session wherein the score isassociated with the likelihood the session participant may take acertain action (e.g., purchase a good or service). Optionally, thecomputing system records/logs related information associated with or ata time of a participant's heart activity change of state. In associationwith a detected participant's heart activity change of state, the systemmay record some or all of the following information: the time and date,a continuous voice snippet/phrase spoken prior to or during the heartactivity change by the participant or other user/participant in thesimulation, other sensory detected attributes associated with theparticipant (e.g., expression, body temperature, perspiration, etc. asfurther described herein), other body and limb action and/or muscularreactions, etc. Optionally, the computing system convertsreceived/recorded speech into text enabling improved data miningincluding search and compare. Optionally, the computing system stores incomputer memory recorded speech and/or converted speech into text inassociation with a user/operator or participant identifier (e.g., a username, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detected heartactivity is determined by the system to change from a configured state.An optional aspect of the computing system enables a user/operator todesignate a heart activity level (e.g., a heart rate range of 80-95beats per minute) of a participant in which monitored feedback may berecorded and/or provided to the user/operator in substantiallyreal-time. Optionally, a simulation is developed with specific heartactivity states in which tracking may be associated (e.g.,automatically). Optionally, the user/operator is presented with a listof heart activity tracking states/ranges prior to or at the launch ofthe simulation and may select a heart activity range and/or state via auser interface control (e.g., a pull down menu of likely heart activityranges and/or states). Optionally, the user/operator may remove saidselected options via a user interface control of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine heart activity baselines, heart rate activity attributes, etc.from which the user's/participant's/subject's heart monitoring istracked throughout the simulation session.

Monitors—Body Temperature

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of body temperature data of a participantcollected using a body temperature sensor. The tracked body temperatureof a participant may be used by itself or in combination with othermonitored factors (e.g., facial expressions) to determine aspects of theparticipant including, for examples, the emotional state, state-of-mind,and/or general well-being of a participant. For example, a detectedand/or determined increased body temperature may indicate a participantmay be under stress in a portion of the simulation in which theincreased body temperature was detected and/or determined.Advantageously, the computing system may derive certain inferences froma body temperature state and/or body temperature state changes (e.g.,reactions to certain simulation events) which may be displayed to auser/operator/observer during and/or subsequent to a virtual realitysimulation. A computing device in association with a body temperaturesensor of a virtual reality participant may record aspects of theparticipant's body temperature and/or any changes and may store theactual sensor recordings to computer memory or data corresponding to theparticipant's sensor output. Advantageously, an aspect of the computingsystem optionally enables an operator or a configured user to receivebody temperature data or inferences corresponding to the bodytemperature data in substantially real-time (e.g., whether theparticipant is relaxed and comfortable) including body temperaturechanges (e.g., an increase in a participant's body temperature) whichmay not be presented in association with an avatar or easily discernableby a configured user and/or operator viewing an avatar representation ofa participant. Optionally, the body temperature sensor may be integratedinto or affixed to a head mounted display of the user. Optionally, thebody temperature sensor may be worn in the form of a strap or othermechanism attached to the participant's body in a manner of recordingbodily temperature. Optionally, body temperature sensor data may bereceived by the computing system wirelessly via a computing device ofthe user/participant (e.g., a smartwatch, smartphone, etc.) Optionally,body temperature sensor data may be received by the computing system viadirect cabling (e.g., a local simulation participant sensor outputconnected to the computing system), via a local area network (e.g., alocal simulation participant sensor output directed to a local computingdevice (e.g., a computing device of the user) connected to the computingsystem over a local area network), via a local wireless network (e.g., alocal simulation participant sensor output to a wireless routerassociated with the computing system), via a wide-area data networkincluding the Internet (e.g., receipt of sensor data from a remoteparticipant's computing device (or a computing device of the computingsystem) wherein the sensor data is wirelessly transmitted to a localcomputing device of the participant). Optionally, the local computingdevice determines certain body temperature-based inferences from thecaptured sensor data output in whole or in part, for example using bodytemperature sensor software analytics, and only transmits said datainferences to the computing system (e.g., via access to shared networkedstorage, sent via a data network, etc.) thereby reducing the amount ofdata sent over the data network from the remote site and so reducingnetwork bandwidth utilization. This approach may reduce the need to havecomplex high performance networks.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant body temperature data in substantially real-time which maynot be provided to the operator/user. An aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a body temperature state dashboard displayable in a portion ofan operator's/user's display (e.g., head mounted display). Optionally,the dashboard displays aspects of the participant's body temperatureand/or certain inferences based upon an analysis of the participant'sbody temperature data in conjunction with aspects of the simulationand/or receipt of other enhanced monitoring data. Optionally, thedisplayed presentation is a representation of a temperature monitor(e.g., a displayed thermometer). Optionally, if there are two or moreparticipant temperature displays, a participant identifier is displayedin association with the temperature displays. An example of a heartactivity inference may be inferring that a participant is pleased inresponse to determining from the sensor that the body temperature iswithin a normal range. A further aspect of the computing system enablesan operator or a configured user to select a participant, via a userinterface control, in which body temperature data may be received.Optionally, the system automatically provides body temperature data toall users/operators/observers, and a user interface control may beoptionally provided whereby each user may disable the presentation ofthe enhanced tracking data. Optionally, the body temperature dashboardis visually presented in a portion of the screen to the user/operatorwith audio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant's body temperature may have increased above athreshold or deviated from a recorded average of the participant).Optionally, received body temperature data is first data analyzed and/orcompared against stored body temperature data or heart activity trackingcues/signals/patterns in order to generating certain inferences asfurther described below before being made available to the operator orconfigured user. Optionally, received body temperature data analysisinvolves accessing historical body temperature data and/or bodytemperature pattern data from one or more internal or external datastores including over wide-area data networks such as the Internet.Optionally, inferences are derived from a comparison of stored bodytemperature state and/or body temperature state changes/patterns fromstored historical simulations, generalized body temperature trackinganalysis (including non-virtual reality simulation environments), and/orother heuristic body temperature tracking research results. A furtheroptional aspect of the system enable the operator and/or user via a setof user interface controls to detect certain body temperature statetarget options or patterns.

With respect to identifying participants in which enhanced bodytemperature monitoring may be received, optionally, a user interfacecontrol may be presented on a display of an operator/user, for examplein response to a control selection by the user/operator. Optionally, inan enhanced body temperature monitoring selection mode state, all theparticipants in a simulation may be presented (e.g., by name, alias,and/or avatar) on the display and a user/operator may select aparticipant presented on the user/operator display. For example, theuser/operator may select a participant via a mouse-like wand control, ahead movement directed at a target user visible on the user/operatordisplay (e.g., by a user/operator centering a session participant in atarget on their screen and then selecting a control), a pull-down listof participants, by speaking the participant's name, using voicecommand(s), or other user interface control selection. Optionally, thecomputing system automatically defaults enhanced body temperaturemonitoring to an operator/administrator (e.g. a user initiating thesimulation, a server user/administrator in a client-serverconfiguration, a local user (versus a remote user), etc.).

With respect to the presentation of body temperature monitoring data ina display format, optionally, a dashboard or window display is projectedon a portion of user/operator display (e.g., a head mounted display,computer monitor, etc.) of the user/operator which may displays certainbody temperature monitoring data. Optionally, the user/operator mayselect a control to expand or shrink the size of the dashboard display.Optionally, the size of the dashboard display may increase or decreasein response to certain conditions detected by the computing system.Optionally, the display dashboard/window size (e.g., in the vertical andhorizontal dimensions, or just in the vertical dimension, or just in thehorizontal dimension) may be automatically increased at least partly inresponse to a determination of a certain pattern of body temperaturewhich may indicate, for example, a certain emotional state or comfortlevel of a participant. Optionally, the display window size may beincreased at least partly in response to detection of certain bodytemperature state changes corresponding to an action of a target.Optionally, the display window size may be increased or decreased atleast partly in response to the occurrence of certain timed events orlack of certain body temperature changes. Optionally, at least partly inresponse to a combination of one or more conditions and/or one or moretimed events, the display window size may be increased or decreased. Forexample, the detection of a subject's body temperature not changing fora fixed period of time may cause the display window size to beincreased. Certain state changes of the simulation itself may bevisually taxing or may require more attention of the user/operator, andin response to detecting the foregoing, the computing system maydecrease the window display size (e.g., automatically). Optionally, thedashboard may display a real-time view, for example, of theparticipant's body temperature as a number and in association with acontinuous graph of body temperature. Optionally, the dashboard may onlydisplay a real-time view of the participant's body temperature activityin response to a condition, such as a body temperature state change.Optionally, a user interface control may be provided which enables theuser to configure the conditions (e.g., an increase or decrease of bodytemperature from a mean body temperature of the participant, bodytemperature greater than a threshold or less than a threshold, etc.)when a view of the participant's body temperature activity may bedisplayed and for how long. Optionally, a user interface control may beprovided which enables the user to instruct that the display window isto be moved to different areas of the monitor. Optionally, the displaywindow may be minimized in response to a user interface controlselection, condition(s), and/or time period expiration. Optionally, bodytemperature monitoring audio cues may be presented to a user whether thedashboard display is minimized or not. Optionally, the user interfacecontrols described herein include but are not limited to one or more ofthe following: a hand gesture (e.g., a hand/limb gesture detected by atracking device), a hand selection by the user/operator of a control(e.g., a touch screen control, a wireless wand control, a computingdevice keyboard, etc.), a voice utterance control (e.g., an audio signalsuch as a whistle, grunt, etc.), a spoken word control (e.g., using aspecific voice command such as “smaller” or “minimize”), a facialgesture (e.g., a wink or two consecutive winks, open mouth, etc.), etc.Optionally, audio/voice control commands are optionally muted inresponse to the user selecting or speaking a mute signal (by speakingthe word “mute”), thereby preventing other simulation participants fromhearing spoken commands from the user. Optionally, the computing systemis configured with a configurable audio communication delay enabling thesystem to detect a control shift keyword (e.g., “Mute”) before streamingspeech from a user/operator over a communication channel with othersimulation participants. Optionally, the system provides a whisper mode,wherein detection of whispered commands or spoken audio below aspecified or determined volume threshold causes the system to preventsuch whispered commands or spoken audio from being transmitted to otherparticipants of the simulation.

Optionally, body temperature monitoring information is presentedsubstantially in real-time to a user/operator in the form of a statusindicator light. For example, if a simulation session participant is ina training session from which enhanced body temperature monitoring datais being received, a small red indicator light may be presented (e.g.,in a portion of the user/operator's display) to a user in response tothe computing system determining that the subject's (e.g. simulationparticipant's) body temperature activity is above a certain range orindicates a certain emotional state, for example, anger or excitement.Optionally, visual indicator cues may be provided by the computingsystem only in response to detecting certain specified body temperaturevariances and/or in the absence of certain body temperature variances.For example, in the use case above, the computing system may display acontinuous visual cue, such as a green light indicator, in response todetermining that a participant's body temperature may indicate theparticipant may be in a relaxed and/or happy state, and a red indicatormay be displayed in response to determining that the participant's bodytemperature may be indicative of a tense or angry state.

With respect to body temperature monitoring audio cues, optionally, anaudio alert may be generated in response to a body temperaturemonitoring state change. For example, in reference to the use caseabove, the system may generate a variance audio tone or whisper when thecomputing system determines a participant's body temperature may havechanged from an indication of a participant's relaxed state to a tenseor angry state. Optionally, a periodic (e.g., every 10 sec, 20 sec, 30sec, 1 min, etc.) compliance audio cue/signal is played to theuser/operator as long as the subject's body temperature may indicate,for example, a relaxed or happy state.

As described herein, the computing system optionally monitors bodytemperature state changes of a participant and subject to certainconfigured constraints/limitations the user/operator is alerted. Forexample, a user/operator may be alerted in substantially real-time inresponse to determining that the participant's body temperature is nolonger in an expected state. For example, it may be determined that aparticipant body temperature are indicative of an unhappy ordisappointed state during a presentation, the user/operator may bealerted to such state of the participant. Optionally, body temperaturemay be continually compared against a knowledge base which may enablethe computing system to infer certain characteristics/attributes of theparticipant's emotional state or state-of-mind. For example, in atraining use case, a low body temperature during a presentation mayinfer the session participant is relaxed and in general agreement withpoints being made by the presenter or may be bored. Alternatively, ahigh body temperature may indicate a session participant is in disbeliefwith aspects of the presentation, excited and/or angry. As the foregoingtwo examples illustrate, the body temperature may be combined with othermonitored activity (e.g., facial expressions) to determine theparticipant's emotional state or state-of-mind (e.g., a high bodytemperature combined with pursed lips may indicate disbelief in certainaspects of a presentation). Optionally, body temperature monitoring maybe recorded during an entire simulation session. Optionally, bodytemperature monitoring may only be recorded during certain visual and/oraudio events and/or conditions (e.g., in response to a state change inthe simulation, an introduction of an element into the simulation, inresponse to a body temperature change of the participant's, etc.)

Optionally, enhanced body temperature monitoring data is recorded,stored, and data mined/analyzed subsequent to the simulation session.Optionally, participants are interviewed subsequent to the simulationsession to determine state-of-mind, reactions, attitudes, physiological,and other psychological criteria to create a knowledge base of bodytemperature in different contextual virtual reality simulations.Optionally, the body temperature readings may be integrated into a datastore to create a knowledge base of body temperature and/or bodytemperature attributes corresponding to a participant's state-of-mindand/or emotion during a simulation or other simulations. This knowledgebase may be used to make certain direct inferences from body temperaturemonitoring data or indirect inferences when body temperature monitoringdata may be combined with other aspects of the simulation (e.g., alikelihood of a product or service sale, expected social media feedback,etc.) Optionally, a created knowledge base from virtual reality enhancedbody temperature monitoring and/or general research related to bodytemperature monitoring may be used to score a session wherein the scoreis associated with the likelihood the session participant may take acertain action (e.g., purchase a good or service). Optionally, thecomputing system records/logs related information associated with or ata time of a participant's body temperature change of state. Inassociation with a detected participant's body temperature change ofstate, the system may record some or all of the following information:the time and date, a continuous voice snippet/phrase spoken prior to orduring the body temperature change by the participant or otheruser/participant in the simulation, other sensory detected attributesassociated with the participant (e.g., expression, heart activity,perspiration, etc. as further described herein), other body and limbaction and/or muscular reactions, etc. Optionally, the computing systemconverts received/recorded speech into text enabling improved datamining including search and compare. Optionally, the computing systemstores in computer memory recorded speech and/or converted speech intotext in association with a user/operator or participant identifier(e.g., a user name, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detected bodytemperature is determined by the system to change from a configuredstate. An optional aspect of the computing system enables auser/operator to designate a body temperature level (e.g., a bodytemperature range of 98-99 degrees Fahrenheit) of a participant in whichmonitored feedback may be recorded and/or provided to the user/operatorin substantially real-time. Optionally, a simulation is developed withspecific body temperature states in which tracking may be associated(e.g., automatically). Optionally, the user/operator is presented with alist of body temperature tracking states/ranges prior to or at thelaunch of the simulation and may select a body temperature point orrange and/or state via a user interface control (e.g., a pull down menuof likely heart activity ranges and/or states). Optionally, theuser/operator may remove said selected options via a user interfacecontrol of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine, for example, body temperature baselines, from which theuser's/participant's/subject's body temperature monitoring is trackedthroughout the simulation session.

Monitors—Perspiration

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of perspiration data of a participantcollected using a wearable perspiration sensor. The amount of and/orcontents of a participant's perspiration tracked during a simulation maybe used by itself or in combination with other monitored factors (e.g.,facial expressions) to determine aspects of the participant including,for examples, the emotional state, state-of-mind, and/or generalwell-being of a participant. Conventionally, for example, a detectedand/or determined increase in the amount of perspiration of aparticipant may indicate the participant is stressed in a portion of thesimulation in which the increased body temperature was detected and/ordetermined. Wearable perspiration detection devices with advancedsensors may be able to determine certain additional and/or more subtlestates and/or state changes in a participant's emotional state via ananalysis of the chemical composition of the received perspiration.Advantageously, the computing system may derive certain inferences fromperspiration state and/or perspiration state changes (e.g., reactions tocertain simulation events) which may be displayed to auser/operator/observer during and/or subsequent to a virtual realitysimulation. A computing device in association with a perspiration sensorof the virtual reality participant may record aspects of theparticipant's perspiration and/or any changes and may store the actualsensor or data corresponding to the participant's sensor output tocomputer memory. Advantageously, an aspect of the computing systemoptionally enables an operator or a configured user to receiveperspiration data or inferences corresponding to the perspiration datain substantially real-time (e.g., whether the participant is relaxed andcomfortable) including perspiration changes (e.g., an increase in aparticipant's perspiration) which may not be presented in associationwith an avatar or easily discernable by a configured user and/oroperator viewing an avatar representation of a participant. Optionally,body perspiration sensor may be integrated into or affixed to a headmounted display of the user. Optionally, the perspiration sensor may beworn in the form of a strap or other mechanism attached to theparticipant's body in a manner of recording bodily perspiration.Optionally, perspiration sensor data may be received by the computingsystem wirelessly via a computing device of the user (e.g., asmartwatch, smartphone, etc.) Optionally, perspiration sensor data maybe received by the computing system via direct cabling (e.g., a localsimulation participant sensor output connected to the computing system),via a local area network (e.g., a local simulation participant sensoroutput directed to a local computing device (e.g., a computing device ofthe user) connected to the computing system over a local area network),via a local wireless network (e.g., a local simulation participantsensor output to a wireless router associated with the computingsystem), via a wide-area data network including the Internet (e.g.,receipt of sensor data from a remote participant's computing device (ora computing device of the computing system) wherein the sensor data iswirelessly transmitted to a local computing device of the participant).Optionally, the local computing device determines certainperspiration-based inferences from the captured sensor data output inwhole or in part, for example using body perspiration sensor softwareanalytics, and only transmits said data analysis to the computing system(e.g., via access to shared networked storage, sent via a data network,etc.) thereby reducing the amount of data sent over the data networkfrom the remote site and so reducing network bandwidth utilization. Thisapproach may reduce the need to have complex high performance networks.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant perspiration data in substantially real-time which may notbe provided to the operator/user. An aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a perspiration state dashboard displayable in a portion of anoperator's/user's display (e.g., head mounted display). Optionally, thedashboard displays aspects of the participant's body perspiration and/orcertain inferences based upon an analysis of the participant'sperspiration data in conjunction with aspects of the simulation and/orreceipt of other enhanced monitoring data. Optionally, the displayedpresentation is a representation of a perspiration monitor (e.g., adisplayed human head with visible perspiration droplets corresponding tothe detected amount of perspiration). Optionally, if there are two ormore participant perspiration monitor displays, a participant identifieris displayed in association with the perspiration monitor displays. Anexample of a perspiration inference may inferring that a participant ispleased in response to determining from the sensor that the detectedperspiration is within a normal range. A further aspect of the computingsystem enables an operator or a configured user to select a participant,via a user interface control, in which perspiration data may bereceived. Optionally, the system automatically provides perspirationdata to all users/operators/observers, and a user interface control maybe optionally provided whereby each user may disable the presentation ofthe enhanced tracking data. Optionally, the perspiration dashboard isvisually presented in a portion of the screen to the user/operator withaudio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant's perspiration may have increased above athreshold or deviated from a recorded average of the participant).Optionally, received perspiration data is first data analyzed and/orcompared against stored body perspiration data or body perspirationtracking cues/signals/patterns in order to generating certain inferencesas further described below before being made available to the operatoror configured user. Optionally, determined and/or received perspirationdata analysis involves accessing historical perspiration data and/orperspiration pattern data from one or more internal or external datastores including over wide-area data networks such as the Internet.Optionally, inferences are derived from a comparison of storedperspiration state changes and/or perspiration state changes/patternsfrom stored historical simulations, generalized perspiration monitoringanalysis (including non-virtual reality simulation environments), and/orother heuristic perspiration monitoring research results. A furtheroptional aspect of the system enable the operator and/or user via a setof user interface controls to detect certain perspiration monitoringtarget options or patterns.

With respect to identifying participants in which enhanced perspirationmonitoring may be received, optionally, a user interface control may bepresented on a display of an operator/user, for example in response to acontrol selection by the user/operator. Optionally, in an enhancedperspiration monitoring selection mode state, all the participants in asimulation may be presented (e.g., by name, alias, and/or avatar) on thedisplay and a user/operator may select a participant presented on theuser/operator display. For example, the user/operator may select aparticipant via a mouse-like wand control, a head movement directed at atarget user visible on the user/operator display (e.g., by auser/operator centering a session participant in a target on theirscreen and then selecting a control), a pull-down list of participants,by speaking the participant's name, using voice command(s), or otheruser interface control selection. Optionally, the computing systemautomatically defaults enhanced perspiration monitoring to anoperator/administrator (e.g. a user initiating the simulation, a serveruser/administrator in a client-server configuration, a local user(versus a remote user), etc.).

With respect to the presentation of perspiration monitoring data in adisplay format, optionally, a dashboard or window display is projectedon a portion of user/operator display (e.g., a head mounted display,computer monitor, etc.) of the user/operator which may displays certainperspiration monitoring data. Optionally, the user/operator may select acontrol to expand or shrink the size of the dashboard display.Optionally, the size of the dashboard display may increase or decreasein response to certain conditions detected by the computing system.Optionally, the display dashboard/window size (e.g., in the vertical andhorizontal dimensions, or just in the vertical dimension, or just in thehorizontal dimension) may be automatically increased at least partly inresponse to a determination of a certain pattern of perspiration whichmay indicate, for example, a certain emotional state or comfort level ofa participant. Optionally, the display window size may be increased atleast partly in response to detection of certain perspiration statechanges corresponding to an action of a target. Optionally, the displaywindow size may be increased or decreased at least partly in response tothe occurrence of certain timed events or lack of certain perspirationchanges. Optionally, at least partly in response to a combination of oneor more conditions and/or one or more timed events, the display windowsize may be increased or decreased. For example, the detection of asubject's perspiration not changing for a fixed period of time may causethe display window size to be increased. Certain state changes of thesimulation itself may be visually taxing or may require more attentionof the user/operator, and in response to detecting the foregoing, thecomputing system may decrease the window display size (e.g.,automatically). Optionally, the dashboard may display a real-time view,for example, of the participant's perspiration as a number and inassociation with a continuous graph of perspiration. Optionally, thedashboard may only display a real-time view of the participant'sperspiration activity in response to a condition, such as a perspirationstate change. Optionally, a user interface control may be provided whichenables the user to configure the conditions (e.g., an increase ordecrease of perspiration from a mean perspiration level of theparticipant, perspiration greater than a threshold or less than athreshold, etc.) when a view of the participant's perspiration activitymay be displayed and for how long. Optionally, a user interface controlmay be provided which enables the user to instruct that the displaywindow is to be moved to different areas of the monitor. Optionally, thedisplay window may be minimized in response to a user interface controlselection, condition(s), and/or time period expiration. Optionally,perspiration monitoring audio cues may be presented to a user whetherthe dashboard display is minimized or not. Optionally, the userinterface controls described herein include but are not limited to oneor more of the following: a hand gesture (e.g., a hand/limb gesturedetected by a tracking device), a hand selection by the user/operator ofa control (e.g., a touch screen control, a wireless wand control, acomputing device keyboard, etc.), a voice utterance control (e.g., anaudio signal such as a whistle, grunt, etc.), a spoken word control(e.g., using a specific voice command such as “smaller” or “minimize”),a facial gesture (e.g., a wink or two consecutive winks, open mouth,etc.), etc. Optionally, audio/voice control commands are optionallymuted in response to the user selecting or speaking a mute signal (byspeaking the word “mute”), thereby preventing other simulationparticipants from hearing spoken commands from the user. Optionally, thecomputing system is configured with a configurable audio communicationdelay enabling the system to detect a control shift keyword (e.g.,“Mute”) before streaming speech from a user/operator over acommunication channel with other simulation participants. Optionally,the system provides a whisper mode, wherein detection of whisperedcommands or spoken audio below a specified or determined volumethreshold causes the system to prevent such whispered commands or spokenaudio from being transmitted to other participants of the simulation.

Optionally, perspiration monitoring information is presentedsubstantially in real-time to a user/operator in the form of a statusindicator light. For example, if a simulation session participant is ina training session from which enhanced perspiration monitoring data isbeing received, a small red indicator light may be presented (e.g., in aportion of the user/operator's display) to a user in response to thecomputing system determining that the subject's (e.g. simulationparticipant's) perspiration activity is above a certain range orindicates a certain emotional state, for example, anger or excitement.Optionally, visual indicator cues may be provided by the computingsystem only in response to detecting certain specified perspirationvariances and/or in the absence of certain perspiration variances. Forexample, in the use case above, the computing system may display acontinuous visual cue, such as a green light indicator, in response todetermining that a participant's perspiration may indicate theparticipant may be in a relaxed and/or happy state, and a red indicatormay be displayed in response to determining that the participant'sperspiration may be indicative of a tense or angry state.

With respect to perspiration monitoring audio cues, optionally, an audioalert may be generated in response to a perspiration monitoring statechange. For example, in reference to the use case above, the system maygenerate a variance audio tone or whisper when the computing systemdetermines a participant's perspiration may have changed from anindication of a participant's relaxed state to a tense or angry state.Optionally, a periodic (e.g., every 10 sec, 20 sec, 30 sec, 1 min, etc.)compliance audio cue/signal is played to the user/operator as long asthe subject's perspiration may indicate, for example, a relaxed or happystate.

As described herein, the computing system optionally monitorsperspiration state changes of a participant and subject to certainconfigured constraints/limitations the user/operator is alerted. Forexample, a user/operator may be alerted in substantially real-time inresponse to determining that the participant's perspiration is no longerin an expected state. For example, it may be determined that aparticipant perspiration are indicative of an unhappy or disappointedstate during a presentation, the user/operator may be alerted to suchstate of the participant. Optionally, perspiration data may becontinually compared against a knowledge base which may enable thecomputing system to infer certain characteristics/attributes of theparticipant's emotional state or state-of-mind. For example, in atraining use case, a low rate of perspiration during a presentation mayinfer the session participant is relaxed and in general agreement withpoints being made by the presenter or may be bored. Alternatively, ahigh rate of perspiration may indicate a session participant is indisbelief with aspects of the presentation, excited, uneasy, tense,and/or angry. As the foregoing two examples illustrate, the perspirationmay be combined with other monitored activity (e.g., facial expressions)to determine the participant's emotional state or state-of-mind (e.g., ahigh perspiration combined with pursed lips may indicate disbelief incertain aspects of a presentation). Optionally, perspiration monitoringmay be recorded during an entire simulation session. Optionally,perspiration monitoring may only be recorded during certain visualand/or audio events and/or conditions (e.g., in response to a statechange in the simulation, an introduction of an element into thesimulation, in response to a change in the participant's perspiration,etc.)

Optionally, enhanced perspiration monitoring data is recorded, stored,and data mined/analyzed subsequent to the simulation session.Optionally, participants are interviewed subsequent to the simulationsession to determine state-of-mind, reactions, attitudes, physiological,and other psychological criteria to create a knowledge base ofperspiration in different contextual virtual reality simulations.Optionally, the perspiration readings may be integrated into a datastore to create a knowledge base of perspiration and/or perspirationattributes corresponding to a participant's state-of-mind and/or emotionduring a simulation or other simulations. This knowledge base may beused to make certain direct inferences from perspiration monitoring dataor indirect inferences when perspiration monitoring data may be combinedwith other aspects of the simulation (e.g., a likelihood of a product orservice sale, expected social media feedback, etc.) Optionally, acreated knowledge base from virtual reality enhanced perspirationmonitoring and/or general research related to perspiration monitoringmay be used to score a session wherein the score is associated with thelikelihood the session participant may take a certain action (e.g.,purchase a good or service). Optionally, the computing systemrecords/logs related information associated with or at a time of aparticipant's perspiration change of state. In association with adetected participant's perspiration change of state, the system mayrecord some or all of the following information: the time and date, acontinuous voice snippet/phrase spoken prior to or during theperspiration change by the participant or other user/participant in thesimulation, other sensory detected attributes associated with theparticipant (e.g., expression, heart activity, perspiration, etc. asfurther described herein), other body and limb action and/or muscularreactions, etc. Optionally, the computing system convertsreceived/recorded speech into text enabling improved data miningincluding search and compare. Optionally, the computing system stores incomputer memory recorded speech and/or converted speech into text inassociation with a user/operator or participant identifier (e.g., a username, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detectedperspiration is determined by the system to change from a configuredstate. An optional aspect of the computing system enables auser/operator to designate a perspiration characteristic (e.g., aperspiration electrolyte content range) of a participant in whichmonitored feedback may be recorded and/or provided to the user/operatorin substantially real-time. Optionally, a simulation is developed withspecific perspiration states in which tracking may be associated (e.g.,automatically). Optionally, the user/operator is presented with a listof perspiration tracking states/ranges prior to or at the launch of thesimulation and may select a perspiration point or range and/or state viaa user interface control (e.g., a pull down menu of likely perspirationrate and/or content ranges and/or states). Optionally, the user/operatormay remove said selected options via a user interface control of thecomputing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine, for example, perspiration rate and/or content baselines, fromwhich the user's/participant's/subject's perspiration monitoring istracked throughout the simulation session.

Monitor—Speech

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of voice data of a participant collectedusing a microphone. The amount, content, inflection, tone, volume,pitch, timing, pauses, energy, etc., hereinafter referred to as speechattributes, of a participant's speech monitored during a simulation maybe used by itself or in combination with other monitored factors (e.g.,facial expressions) to determine aspects of the participant including,for examples, the emotional state (e.g., mood), state-of-mind, and/orgeneral well-being of a participant. Just as a person may be able todistinguish the emotional state of another person, speech analysis usingcomputer methods and algorithms may be able to similarly distinguishsaid state of a person speaking. Optionally, speech analysis may be usedto determine the truthfulness of certain content spoken by a user.Optionally, recorded speech of a participant is converted into text(e.g. speech-to-text). Advantageously, an aspect of the computing systemoptionally enables an operator or a configured user to receive speechanalysis or speech inferences corresponding to the speech data insubstantially real-time (e.g., whether the participant is relaxed andcomfortable) including speech analysis changes (e.g., an increase in aparticipant's perspiration) which may not be presented in associationwith an avatar or easily discernable by a configured user and/oroperator viewing an avatar representation of a participant.

Advantageously, the computing system may derive certain inferences fromthe speech content and/or speech attributes recorded during a virtualreality simulation which may be displayed to a user/operator/observerduring and subsequent to the simulation. A computing device including amicrophone in association with the virtual reality participant mayrecord the speech and/or aspects of the participant's speech and maystore the actual recorded speech or aspects of the participant's speechto computer memory. Optionally, a microphone is integrated into oraffixed to a head mounted display of the user. Optionally, a microphonemay be worn in the form of a strap or other mechanism attached to theparticipant's body in a manner of recording speech. Optionally, recordedspeech may be received by the computing system via direct cabling (e.g.,a local simulation participant's microphone connected to the computingsystem), via a local area network (e.g., a local simulation participantvoice output directed to a local computing device (e.g., a computingdevice of the user) connected to the computing system over a local areanetwork), via a local wireless network (e.g., a local simulationparticipant microphone output to a wireless router associated with thecomputing system), via a wide-area data network including the Internet(e.g., receipt of voice data from a remote participant's computingdevice (or a computing device of the computing system) wherein thesensor data is wirelessly transmitted to a local computing device of theparticipant). Optionally, recorded speech is converted to text prior totransmission from a remote location in order to reduce the data networkdemands and latencies. Optionally, the local computing device determinesfrom the microphone output in whole or in part certain speechattributes/characteristics, for example using speech software analytics,and only transmits said data analysis to the computing system (e.g., viaaccess to shared networked storage, sent via a data network, etc.)thereby reducing the amount of data sent over the data network from theremote site and so reducing network bandwidth utilization. This approachmay reduce the need to have complex high performance networks.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant perspiration data in substantially real-time which may notbe provided to the operator/user. An aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a voice state dashboard displayable in a portion of anoperator's/user's display (e.g., head mounted display). Optionally, thedashboard displays aspects of the participant's speech based upon ananalysis of the participant's voice data in conjunction with aspects ofthe simulation and/or other enhanced monitoring data. A further aspectof the computing system enables an operator or a configured user toselect a participant, via a user interface control, in which voice datainferences may be received. Optionally, the system automaticallyprovides voice data and/or voice inferences to allusers/operators/observers, and a user interface control may beoptionally provided whereby each user may disable the presentation ofthe enhanced monitoring data. Optionally, the voice analysis dashboardis visually presented in a portion of the screen to the user/operatorwith audio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant's tone or keyword content may indicate aparticipant's likelihood of purchasing a good or service). Optionally,received voice data is first data analyzed and/or compared againststored speech analysis data or speech tracking cues/signals/patterns inorder to generating certain inferences as further described below beforebeing made available to the operator or configured user. Optionally,determined and/or received speech data/analysis involves accessinghistorical data and/or historical speech pattern data from one or moreinternal or external data stores including over wide-area data networkssuch as the Internet. Optionally, inferences are derived from acomparison of stored voice state changes and/or speech changes/patternsfrom stored historical simulations, generalized speech monitoringanalysis (including non-virtual reality simulation environments), and/orother heuristic speech monitoring research results. A further optionalaspect of the system enable the operator and/or user via a set of userinterface controls to configure certain speech monitoring target optionsor patterns.

With respect to identifying participants in which enhanced speechmonitoring may be received, optionally, a user interface control may bepresented on a display of an operator/user, for example in response to acontrol selection by the user/operator. Optionally, in an enhancedspeech monitoring selection mode state, all the participants in asimulation may be presented (e.g., by name, alias, and/or avatar) on thedisplay and a user/operator may select a participant presented on theuser/operator display. For example, the user/operator may select aparticipant via a mouse-like wand control, a head movement directed at atarget user visible on the user/operator display (e.g., by auser/operator centering a session participant in a target on theirscreen and then selecting a control), a pull-down list of participants,by speaking the participant's name, using voice command(s), or otheruser interface control selection. Optionally, the computing systemautomatically defaults enhanced perspiration monitoring to anoperator/administrator (e.g. a user initiating the simulation, a serveruser/administrator in a client-server configuration, a local user(versus a remote user), etc.).

With respect to the presentation of speech monitoring data in a displayformat, optionally, a dashboard or window display is projected on aportion of user/operator display (e.g., a head mounted display, computermonitor, etc.) of the user/operator which may displays certain speechmonitoring data. Optionally, the user/operator may select a control toexpand or shrink the size of the dashboard display. Optionally, the sizeof the dashboard display may increase or decrease in response to certainconditions detected by the computing system. Optionally, the displaydashboard/window size (e.g., in the vertical and horizontal dimensions,or just in the vertical dimension, or just in the horizontal dimension)may be automatically increased at least partly in response to adetermination of a certain pattern of speech which may indicate, forexample, a certain emotional state or comfort level of a participant.Optionally, the display window size may be increased at least partly inresponse to detection of certain speech state changes in conjunctionwith corresponding to an action of a target. Optionally, the displaywindow size may be increased or decreased at least partly in response tothe occurrence of certain timed events or lack speech by a participant.Optionally, at least partly in response to a combination of one or moreconditions and/or one or more timed events, the display window size maybe increased or decreased. For example, the detection of a subject's notspeaking for a fixed period of time may cause the display window size tobe increased. Certain state changes of the simulation itself may bevisually taxing or may require more attention of the user/operator, andin response to detecting the foregoing, the computing system maydecrease the window display size (e.g., automatically). Optionally, thedashboard may display a real-time view, for example, of theparticipant's speech as a number and in association with a continuousgraph of speech. Optionally, the dashboard may only display a real-timeview of the participant's speech activity in response to a condition,such as a speech state change. Optionally, a user interface control maybe provided which enables the user to configure the conditions (e.g., anincrease or decrease of speech from a mean speech level of theparticipant, speech greater than a threshold or less than a threshold,etc.) when a view of the participant's speech activity may be displayedand for how long. Optionally, a user interface control may be providedwhich enables the user to instruct that the display window is to bemoved to different areas of the monitor. Optionally, the display windowmay be minimized in response to a user interface control selection,condition(s), and/or time period expiration. Optionally, speechmonitoring audio cues may be presented to a user whether the dashboarddisplay is minimized or not. Optionally, the user interface controlsdescribed herein include but are not limited to one or more of thefollowing: a hand gesture (e.g., a hand/limb gesture detected by atracking device), a hand selection by the user/operator of a control(e.g., a touch screen control, a wireless wand control, a computingdevice keyboard, etc.), a voice utterance control (e.g., an audio signalsuch as a whistle, grunt, etc.), a spoken word control (e.g., using aspecific voice command such as “smaller” or “minimize”), a facialgesture (e.g., a wink or two consecutive winks, open mouth, etc.), etc.Optionally, audio/voice control commands are optionally muted inresponse to the user selecting or speaking a mute signal (by speakingthe word “mute”), thereby preventing other simulation participants fromhearing spoken commands from the user. Optionally, the computing systemis configured with a configurable audio communication delay enabling thesystem to detect a control shift keyword (e.g., “Mute”) before streamingspeech from a user/operator over a communication channel with othersimulation participants. Optionally, the system provides a whisper mode,wherein detection of whispered commands or spoken audio below aspecified or determined volume threshold causes the system to preventsuch whispered commands or spoken audio from being transmitted to otherparticipants of the simulation.

Optionally, speech monitoring information is presented substantially inreal-time to a user/operator in the form of a status indicator light.For example, if a simulation session participant is in a trainingsession from which enhanced speech monitoring data is being received, asmall red indicator light may be presented (e.g., in a portion of theuser/operator's display) to a user in response to the computing systemdetermining that the subject's (e.g. simulation participant's) speechactivity is indicative of a certain condition such as an emotional state(e.g., anger). Optionally, visual indicator cues may be provided by thecomputing system only in response to detecting certain specified speechvariances and/or in the absence of certain speech variances. Forexample, in the use case above, the computing system may display acontinuous visual cue, such as a green light indicator, in response todetermining that a participant's speech may indicate the participant maybe in a relaxed and/or happy state, and a red indicator may be displayedin response to determining that the participant's speech may beindicative of a tense or angry state.

With respect to speech monitoring audio cues, optionally, an audio alertmay be generated in response to a speech monitoring state change. Forexample, in reference to the use case above, the system may generate avariance audio tone or whisper when the computing system determines aparticipant's speech may have changed from an indication of aparticipant's relaxed state to a tense or angry state. Optionally, aperiodic (e.g., every 10 sec, 20 sec, 30 sec, 1 min, etc.) complianceaudio cue/signal is played to the user/operator as long as the subject'sspeech may indicate, for example, a relaxed or happy state.

As described herein, the computing system optionally monitors speechstate changes of a participant and subject to certain configuredconstraints/limitations the user/operator is alerted. For example, auser/operator may be alerted in substantially real-time in response todetermining that the participant's speech is no longer in an expectedstate. For example, it may be determined that a participant speech areindicative of an unhappy or disappointed state during a presentation,the user/operator may be alerted to such state of the participant.Optionally, speech data may be continually compared against a knowledgebase which may enable the computing system to infer certaincharacteristics/attributes of the participant's emotional state orstate-of-mind. For example, in a training use case, certain verbal cues(e.g., yes) during a presentation may infer the session participant isrelaxed and in general agreement with points being made by thepresenter. Speech may be combined with other monitored activity (e.g.,facial expressions) to determine the participant's emotional state orstate-of-mind (e.g., negative verbal cues combined with pursed lips mayindicate disbelief in certain aspects of a presentation). Optionally,all speech of a participant may be recorded during a simulation session.Optionally, speech monitoring may only be recorded during certain visualand/or audio events and/or conditions (e.g., in response to a statechange in the simulation, an introduction of an element into thesimulation, in response to a change in a participant's speech, etc.)

Optionally, enhanced speech monitoring data is recorded, stored, anddata mined/analyzed subsequent to the simulation session. Optionally,participants are interviewed subsequent to the simulation session todetermine state-of-mind, reactions, attitudes, physiological, and otherpsychological criteria to create a knowledge base of speech in differentcontextual virtual reality simulations. Optionally, the speech readingsmay be integrated into a data store to create a knowledge base of speechand/or speech attributes corresponding to a participant's state-of-mindand/or emotion during a simulation or other simulations. This knowledgebase may be used to make certain direct inferences from speechmonitoring data or indirect inferences when speech monitoring data maybe combined with other aspects of the simulation (e.g., a likelihood ofa product or service sale, expected social media feedback, etc.)Optionally, a created knowledge base from virtual reality enhancedspeech monitoring and/or general speech analysis research may be used toscore a session wherein the score is associated with the likelihood thesession participant may take a certain action (e.g., purchase a good orservice). Optionally, the computing system records/logs relatedinformation associated with or at a time of a participant's speech(e.g., content) or speech change of state. In association with adetected participant's speech or certain speech change of state, thesystem may record some or all of the following information: the time anddate, a continuous voice snippet/phrase spoken prior to or during thespeech change by the participant or other user/participant in thesimulation, other sensory detected attributes associated with theparticipant (e.g., expression, heart activity, speech, etc. as furtherdescribed herein), other body and limb action and/or muscular reactions,etc. Optionally, the computing system converts received/recorded speechinto text enabling improved data mining including search and compare.Optionally, the computing system stores in computer memory recordedspeech and/or converted speech into text in association with auser/operator or participant identifier (e.g., a user name, accountnumber, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detected speechis determined by the system to change from a configured state. Anoptional aspect of the computing system enables a user/operator todesignate a speech characteristic (e.g., anger tone) of a participant inwhich monitored feedback may be recorded and/or provided to theuser/operator in substantially real-time. Optionally, a simulation isdeveloped with specific speech states in which tracking may beassociated (e.g., automatically). Optionally, the user/operator ispresented with a list of speech tracking states/ranges prior to or atthe launch of the simulation and may select a spoken phrase and/or voiceanalysis state (e.g., angry) and/or detected speech analysis statechange via a user interface control (e.g., a pull down menu of likelyverbal cues). Optionally, the user/operator may remove said selectedoptions via a user interface control of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine, for example, speech rate and/or content baselines, from whichthe user's/participant's/subject's speech monitoring is trackedthroughout the simulation session.

Monitor Breath

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of breath content data of a participantcollected using a sensor, for example, a breath analyzer. The volume(e.g., deep or shallow), rate, pattern, and content of a participant'sbreath tracked during a simulation may be used by itself or incombination with other monitored factors (e.g., facial expressions) todetermine aspects of the participant including, for examples, theemotional state, state-of-mind, and/or general well-being of aparticipant. Conventionally, for example, rapid or increased breathingof a user may be a factor indicating increased excitement and/or stressin a portion of the simulation in which the increased breathing rate isdetected and/or determined. In another example, the breath content of auser may be analyzed to determine the likelihood of certain diseasessuch as cancer. Optionally, detection of certain tracemolecules/chemicals in a breath may indicate an increase in stress forexample. Wearable breath sensors may be able to determine certainadditional and/or more subtle states and/or state changes in aparticipant's emotional state via an analysis of the molecular/chemicalcomposition of a received exhale. Advantageously, the computing systemmay derive certain inferences from a breath's state changes (e.g.,reactions to certain simulation events) which may be displayed to auser/operator/observer during and/or subsequent to a virtual realitysimulation. A computing device in association with a breath sensor ofthe virtual reality participant may record aspects of the participant'sbreath content, rate, volume, and/or any changes and may store theactual sensor or data corresponding to the participant's sensor outputto computer memory. Advantageously, an aspect of the computing systemoptionally enables an operator or a configured user to receive breathsensor data or inferences corresponding to the breath data insubstantially real-time (e.g., whether the participant is relaxed andcomfortable) including breath changes (e.g., an increase in aparticipant's breathing rate) which may not be presented in associationwith an avatar or easily discernable by a configured user and/oroperator viewing an avatar representation of a participant. Optionally,breath sensor may be integrated into or affixed to a head mounteddisplay of the user. Optionally, breath exhale sensor data may bereceived by the computing system wirelessly via a computing device ofthe user (e.g., a smartwatch, smartphone, etc.) Optionally, a breathsensor is portable and may be worn in, placed near, and/or placed in aparticipant's mouth for the duration of the simulation or a portion ofthe simulation. Optionally, breath sensor data may be received by thecomputing system via direct cabling (e.g., a local simulationparticipant sensor output connected to the computing system), via alocal area network (e.g., a local simulation participant sensor outputdirected to a local computing device (e.g., a computing device of theuser) connected to the computing system over a local area network), viaa local wireless network (e.g., a local simulation participant sensoroutput to a wireless router associated with the computing system), via awide-area data network including the Internet (e.g., receipt of sensordata from a remote participant's computing device (or a computing deviceof the computing system) wherein the sensor data is wirelesslytransmitted to the computing device of the participant). Optionally, thelocal computing device determines certain breath-based inferences fromthe captured sensor data output in whole or in part, for example usingbreath sensor software analytics, and only transmits said data analysisand/or breath attributes to the computing system (e.g., via access toshared networked storage, sent via a data network, etc.) therebyreducing the amount of data sent over the data network from the remotesite and so reducing network bandwidth utilization. This approach mayreduce the need to have complex high performance networks.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant breath data in substantially real-time which may not beprovided to the operator/user. An aspect of the computing system enablesan operator or a configured user to be presented with, for example, abreath state dashboard displayable in a portion of an operator's/user'sdisplay (e.g., head mounted display). Optionally, the dashboard displaysaspects of the participant's breath and/or certain inferences based uponan analysis of the participant's breath data in conjunction with aspectsof the simulation and/or receipt of other enhanced monitoring data.Optionally, the dashboard display is a representation of a designatedparticipant's breathing rate (e.g., a sine wave graph corresponding tothe breathing pattern of the participant). Optionally, if there are twoor more participant breath displays, a participant identifier isdisplayed in association with the breath displays. An example of abreath inference may be inferring that a participant is pleased inresponse to determining from the sensor that the detected breathing rateis within a normal range for the participant. A further aspect of thecomputing system enables an operator or a configured user to select aparticipant, via a user interface control, in which breath exhale datamay be received. Optionally, the system automatically provides breathexhale data to all users/operators/observers, and a user interfacecontrol may be optionally provided whereby each user may disable thepresentation of the enhanced tracking data. Optionally, the breathexhale dashboard is visually presented in a portion of the screen to theuser/operator with audio cues or exclusively via audio cues (e.g., usingan audio tone indicating a participant's breath may have increased abovea threshold or deviated from a recorded average of the participant).Optionally, received breath exhale data is first data analyzed and/orcompared against stored breath data or breath trackingcues/signals/patterns in order to generating certain inferences asfurther described below before being made available to the operator orconfigured user. Optionally, the determined and/or received breath dataanalysis involves accessing historical breath data and/or breath patterndata from one or more internal or external data stores including overwide-area data networks such as the Internet. Optionally, inferences arederived from a comparison of stored breath state changes and/or breathstate changes/patterns from stored historical simulations, generalizedbreath exhale monitoring analysis (including non-virtual realitysimulation environments), and/or other heuristic breath exhalemonitoring research results. A further optional aspect of the systemenable the operator and/or user via a set of user interface controls todetect certain breath monitoring target options.

With respect to identifying participants in which enhanced breathmonitoring may be received, optionally, a user interface control may bepresented on a display of an operator/user, for example in response to acontrol selection by the user/operator. Optionally, in an enhancedbreath monitoring selection mode state, all the participants in asimulation may be presented (e.g., by name, alias, and/or avatar) on thedisplay and a user/operator may select a participant presented on theuser/operator display. For example, the user/operator may select aparticipant via a mouse-like wand control, a head movement directed at atarget user visible on the user/operator display (e.g., by auser/operator centering a session participant in a target on theirscreen and then selecting a control), a pull-down list of participants,by speaking the participant's name, using voice command(s), or otheruser interface control selection. Optionally, the computing systemautomatically defaults enhanced breath monitoring to anoperator/administrator (e.g. a user initiating the simulation, a serveruser/administrator in a client-server configuration, a local user(versus a remote user), etc.).

With respect to the presentation of breath monitoring data in a displayformat, optionally, a dashboard or window display is projected on aportion of user/operator display (e.g., a head mounted display, computermonitor, etc.) of the user/operator which may displays certain breathmonitoring data. Optionally, the user/operator may select a control toexpand or shrink the size of the dashboard display. Optionally, the sizeof the dashboard display may increase or decrease in response to certainconditions detected by the computing system. Optionally, for example,the display dashboard/window size (e.g., in the vertical and horizontaldimensions, or just in the vertical dimension, or just in the horizontaldimension) may be automatically increased at least partly in response toa determination of a certain pattern of breath which may indicate, forexample, a certain emotional state or comfort level of a participant.Optionally, the display window size may be increased at least partly inresponse to detection of certain breath state changes (e.g., content orrate) corresponding to an action of a designated target. Optionally, thedisplay window size may be increased or decreased at least partly inresponse to the occurrence of certain timed events or lack of certainbreath changes. Optionally, at least partly in response to a combinationof one or more conditions and/or one or more timed events, the displaywindow size may be increased or decreased. For example, the detection ofa subject's breath not changing for a fixed period of time may cause thedisplay window size to be increased. Certain state changes of thesimulation itself may be visually taxing or may require more attentionof the user/operator, and in response to detecting the foregoing, thecomputing system may decrease the window display size (e.g.,automatically). Optionally, the dashboard may display a real-time view,for example, of the participant's breath rate as a number (e.g., 20breaths per minute) and in association with a continuous graph of breathactivity. Optionally, the dashboard may only display a real-time view ofthe participant's breath activity in response to a condition, such as abreath state change. Optionally, a user interface control may beprovided which enables the user to configure the conditions (e.g., anincrease or decrease of breath rate from a mean breath rate of theparticipant, breath rate greater than a threshold or less than athreshold, etc.) when a view of the participant's breath activity may bedisplayed and for how long. Optionally, a user interface control may beprovided which enables the user to instruct that the display window isto be moved to different areas of the monitor. Optionally, the displaywindow may be minimized in response to a user interface controlselection, condition(s), and/or time period expiration. Optionally,breath monitoring audio cues may be presented to a user whether thedashboard display is minimized or not. Optionally, the user interfacecontrols described herein include but are not limited to one or more ofthe following: a hand gesture (e.g., a hand/limb gesture detected by atracking device), a hand selection by the user/operator of a control(e.g., a touch screen control, a wireless wand control, a computingdevice keyboard, etc.), a voice utterance control (e.g., an audio signalsuch as a whistle, grunt, etc.), a spoken word control (e.g., using aspecific voice command such as “smaller” or “minimize”), a facialgesture (e.g., a wink or two consecutive winks, open mouth, etc.), etc.Optionally, audio/voice control commands are optionally muted inresponse to the user selecting or speaking a mute signal (by speakingthe word “mute”), thereby preventing other simulation participants fromhearing spoken commands from the user. Optionally, the computing systemis configured with a configurable audio communication delay enabling thesystem to detect a control shift keyword (e.g., “Mute”) before streamingspeech from a user/operator over a communication channel with othersimulation participants. Optionally, the system provides a whisper mode,wherein detection of whispered commands or spoken audio below aspecified or determined volume threshold causes the system to preventsuch whispered commands or spoken audio from being transmitted to otherparticipants of the simulation.

Optionally, breath monitoring information is presented substantially inreal-time to a user/operator in the form of a status indicator light.For example, if a simulation session participant is in a trainingsession from which enhanced breath monitoring data is being received, asmall red indicator light may be presented (e.g., in a portion of theuser/operator's display) to a user in response to the computing systemdetermining that the subject's (e.g. simulation participant's) breathactivity is above a certain range or indicates a certain emotionalstate, for example, anger or excitement. Optionally, visual indicatorcues may be provided by the computing system only in response todetecting certain specified breath variances and/or in the absence ofcertain breath variances. For example, in the use case above, thecomputing system may display a continuous visual cue, such as a greenlight indicator, in response to determining that a participant's breathmay indicate the participant may be in a relaxed and/or happy state, anda red indicator may be displayed in response to determining that theparticipant's breath may be indicative of a tense or angry state.

With respect to breath monitoring audio cues, optionally, an audio alertmay be generated in response to a breath monitoring state change. Forexample, in reference to the use case above, the system may generate avariance audio tone or whisper when the computing system determines aparticipant's breath activity may have changed from an indication of aparticipant's relaxed state to a tense or angry state. Optionally, aperiodic (e.g., every 10 sec, 20 sec, 30 sec, 1 min, etc.) complianceaudio cue/signal is played to the user/operator as long as the subject'sbreath may indicate, for example, a relaxed or happy state.

As described herein, the computing system optionally monitors breathstate changes of a participant and subject to certain configuredconstraints/limitations the user/operator is alerted. For example, auser/operator may be alerted in substantially real-time in response todetermining that the participant's breath is no longer in an expectedstate. For example, it may be determined that a participant breath areindicative of an unhappy or disappointed state during a presentation,the user/operator may be alerted to such state of the participant.Optionally, breath data may be continually compared against a knowledgebase which may enable the computing system to infer certaincharacteristics/attributes of the participant's emotional state orstate-of-mind. For example, in a training use case, a low rate of breathactivity during a presentation may infer the session participant isrelaxed and in general agreement with points being made by the presenteror may be bored. Alternatively, a high rate of breath activity mayindicate a session participant is in disbelief with aspects of thepresentation, excited, uneasy, tense, and/or angry. As the foregoing twoexamples illustrate, the breath data may be combined with othermonitored activity (e.g., facial expressions) to determine theparticipant's emotional state or state-of-mind (e.g., a high breath ratecombined with pursed lips may indicate disbelief in certain aspects of apresentation). Optionally, breath monitoring may be recorded during anentire simulation session. Optionally, breath monitoring may only berecorded during certain visual and/or audio events and/or conditions(e.g., in response to a state change in the simulation, an introductionof an element into the simulation, in response to a participant's breathchange of state, etc.)

Optionally, enhanced breath monitoring data is recorded, stored, anddata mined/analyzed subsequent to the simulation session. Optionally,participants are interviewed subsequent to the simulation session todetermine state-of-mind, reactions, attitudes, physiological, and otherpsychological criteria to create a knowledge base of breath data indifferent contextual virtual reality simulations. Optionally, the breathreadings may be integrated into a data store to create a knowledge baseof breathing and/or breath attributes corresponding to a participant'sstate-of-mind and/or emotion during a simulation or other simulations.This knowledge base may be used to make certain direct inferences frombreath monitoring data or indirect inferences when breath monitoringdata may be combined with other aspects of the simulation (e.g., alikelihood of a product or service sale, expected social media feedback,etc.) Optionally, a created knowledge base from virtual reality enhancedbreath monitoring and/or general research related to breath monitoringmay be used to score a session wherein the score is associated with thelikelihood the session participant may take a certain action (e.g.,purchase a good or service). Optionally, the computing systemrecords/logs related information associated with or at a time of aparticipant's breath state changes. In association with a detectedparticipant's breath state change, the system may record some or all ofthe following information: the time and date, a continuous voicesnippet/phrase spoken prior to or during the breath change by theparticipant or other user/participant in the simulation, other sensorydetected attributes associated with the participant (e.g., expression,heart activity, perspiration, etc. as further described herein), otherbody and limb action and/or muscular reactions, etc. Optionally, thecomputing system converts received/recorded speech into text enablingimproved data mining including search and compare. Optionally, thecomputing system stores in computer memory recorded speech and/orconverted speech into text in association with a user/operator orparticipant identifier (e.g., a user name, account number, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detected breathis determined by the system to change from a configured state. Anoptional aspect of the computing system enables a user/operator todesignate a breath characteristic (e.g., a breath trace element/chemicalcompound content range) of a participant in which monitored feedback maybe recorded and/or provided to the user/operator in substantiallyreal-time. Optionally, a simulation is developed with specific breathstates in which tracking may be associated (e.g., automatically).Optionally, the user/operator is presented with a list of breathtracking states/ranges prior to or at the launch of the simulation andmay select a breath point or range and/or state via a user interfacecontrol (e.g., a pull down menu of likely breath rates and/or contentranges and/or states). Optionally, the user/operator may remove saidselected options via a user interface control of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine, for example, breath rate and/or content baselines, from whichthe user's/participant's/subject's breath monitoring is trackedthroughout the simulation session.

Monitor Hand/Limb

An optional aspect of an interactive, multi-user, virtual realitysimulation is the utilization of hand/limb activity which may becollected using a sensor such as a tracking device and/or haptic glovetracking mechanism including a glove affixed with passive or activemarkers, and/or wearable passive or active markers. Hand/limb activity(e.g., clenched fist, hand force applied to a wand, finger pointing,scratching, nail biting, etc.) tracked during a simulation may be usedby itself or in combination with other monitored factors (e.g., facialexpressions) to determine aspects of the participant including, forexamples, the emotional state, state-of-mind, and/or general well-beingof a simulation participant. Conventionally, for example, a detectedand/or determined clenched fist may be a factor indicating increasedexcitement and/or stress. A computing device in association with ahand/limb sensor and/or tracking device of the virtual realityparticipant may record aspects of the participant's perspiration and/orany changes and may store the actual sensor or data corresponding to theparticipant's sensor output and/or tracking device to computer memory.Advantageously, the computing system may derive certain inferences froma hand/limb activity and/or hand/limb activity state changes (e.g., inreactions to certain simulation events) which may be displayed to auser/operator/observer during and/or subsequent to a virtual realitysimulation. Advantageously, an aspect of the computing system optionallyenables an operator or a configured user to receive additional hand/limbactivity data or inferences corresponding to the hand/limb activity datain substantially real-time (e.g., whether the participant is relaxed andcomfortable) including hand/limb activity changes (e.g., a certain typeof hand action such as a hand wave) which may not be easily discernableby a user/operator viewing an avatar representation of a participant. Acomputing device in association with a hand/limb activity sensor ortracking device of the virtual reality participant may record theparticipant's hand/limb activity and/or any changes and store saidrecordings to computer memory. Optionally, a hand/limb activity sensoris worn in the form of a glove or other mechanism attached to theparticipant's hand in a manner enabling the recording hand activity Anexample hand-held tracking device which emits trackable infrared lightsignatures is a wireless Precision Position Tracking (PPT) device, PPTEyes™, from WorldViz LLC of Santa Barbara. The wand, used in conjunctionwith motion tracking cameras, provides interaction and navigation inimmersive virtual reality systems and may include a joystick/pad andtrigger enabling hand tracking and hand grabbing and/or moving ofvirtual objects.

Optionally, hand action data may be received by the computing systemwirelessly via a computing device of the user (e.g., a smartwatch,smartphone, etc.) Optionally, hand/limb activity sensor/tracking data isreceived by the computing system via direct cabling (e.g., a localsimulation participant sensor output connected to the computing system),via a local area network (e.g., a local simulation participant sensoroutput directed to a local computing device (e.g., a computing device ofthe user) connected to the computing system over a local area network),via a local wireless network (e.g., a local simulation participantsensor output to a wireless router associated with the computingsystem), via a wide-area data network including the Internet (e.g.,receipt of sensor data from a remote participant's computing device (ora computing device of the computing system) wherein the sensor data iswirelessly transmitted to a local computing device of the participant).Optionally, the local computing device determines certain hand/limbactivity-based inferences from the captured sensor/tracking data outputin whole or in part, for example using hand/limb activitysensor/tracking software analytics, and only transmits said dataanalysis to the computing system (e.g., via access to shared networkedstorage, sent via a data network, etc.) thereby reducing the amount ofdata sent over the data network from the remote site and so reducingnetwork bandwidth utilization. This approach may reduce the need to havecomplex high performance networks.

As similarly described above with respect to head tracking, an aspect ofthe computing system enables an operator/configured user to receiveparticipant hand/limb activity data in substantially real-time which maynot be provided to the operator/user. An aspect of the computing systemenables an operator or a configured user to be presented with, forexample, a hand/limb activity state dashboard displayable in a portionof an operator's/user's display (e.g., head mounted display).Optionally, the dashboard displays aspects of the participant'shand/limb activity and/or certain inferences based upon an analysis ofthe participant's hand/limb activity data in conjunction with aspects ofthe simulation and/or receipt of other enhanced monitoring data.Optionally, the dashboard display is a representation of a designatedparticipant's hand/limb activity. Optionally, if there are two or moreparticipant breath displays, a participant identifier is displayed inassociation with the breath displays. An example of a hand/limb activityinference is inferring that a participant is pleased in response to thecomputing system determining that he participant signaled a “thumbs up”.A further aspect of the computing system enables an operator or aconfigured user to select a participant, via a user interface control,in which hand action data may be received. Optionally, the systemautomatically provides hand action data to allusers/operators/observers, and a user interface control may beoptionally provided whereby each user may disable the presentation ofthe enhanced tracking data. Optionally, the hand/limb activity dashboardis visually presented in a portion of the screen to the user/operatorwith audio cues or exclusively via audio cues (e.g., using an audio toneindicating a participant's hand/limb activity may have increased above athreshold or deviated from a recorded average of the participant).Optionally, received hand/limb activity data is first data analyzedand/or compared against stored hand/limb activity data or hand/limbactivity tracking cues/signals/patterns in order to generating certaininferences as further described below before being made available to theoperator or configured user. Optionally, the determined and/or receivedhand/limb activity data analysis involves accessing historical hand/limbactivity and/or hand/limb activity pattern data from one or moreinternal or external data stores including over wide-area data networkssuch as the Internet. Optionally, inferences are derived from acomparison of stored hand/limb activity state changes and/or hand/limbactivity state changes/patterns from stored historical simulations,generalized hand action monitoring analysis (including non-virtualreality simulation environments), and/or other heuristic hand actionmonitoring research results. A further optional aspect of the systemenable the operator and/or user via a set of user interface controls toconfigure certain hand action monitoring target options or patterns.

With respect to identifying participants in which enhanced hand/limbactivity monitoring may be received, optionally, a user interfacecontrol may be presented on a display of an operator/user, for examplein response to a control selection by the user/operator. Optionally, inan enhanced hand/limb activity monitoring selection mode state, all theparticipants in a simulation may be presented (e.g., by name, alias,and/or avatar) on the display and a user/operator may select aparticipant presented on the user/operator display. For example, theuser/operator may select a participant via a mouse-like wand control, ahead movement directed at a target user visible on the user/operatordisplay (e.g., by a user/operator centering a session participant in atarget on their screen and then selecting a control), a pull-down listof participants, by speaking the participant's name, using voicecommand(s), or other user interface control selection. Optionally, thecomputing system automatically defaults enhanced hand/limb activitymonitoring to an operator/administrator (e.g. a user initiating thesimulation, a server user/administrator in a client-serverconfiguration, a local user (versus a remote user), etc.).

With respect to the presentation of hand/limb activity monitoring datain a display format, optionally, a dashboard or window display isprojected on a portion of user/operator display (e.g., a head mounteddisplay, computer monitor, etc.) of the user/operator which may displayscertain hand/limb activity monitoring data. Optionally, theuser/operator may select a control to expand or shrink the size of thedashboard display. Optionally, the size of the dashboard display mayincrease or decrease in response to certain conditions detected by thecomputing system. Optionally, the display dashboard/window size (e.g.,in the vertical and horizontal dimensions, or just in the verticaldimension, or just in the horizontal dimension) may be automaticallyincreased at least partly in response to a determination of a certainpattern of hand/limb activity which may indicate, for example, a certainemotional state or comfort level of a participant. Optionally, thedisplay window size may be increased at least partly in response todetection of certain hand/limb activity state changes corresponding toan action of a designated target. Optionally, the display window sizemay be increased or decreased at least partly in response to theoccurrence of certain timed events or lack of certain hand/limb activitychanges. Optionally, at least partly in response to a combination of oneor more conditions and/or one or more timed events, the display windowsize may be increased or decreased. For example, the detection of asubject's hand/limb activity not changing for a fixed period of time maycause the display window size to be increased. Certain state changes ofthe simulation itself may be visually taxing or may require moreattention of the user/operator, and in response to detecting theforegoing, the computing system may decrease the window display size(e.g., automatically). Optionally, the dashboard may display a real-timeview, for example, of the participant's hand/limb or a virtualrepresentation mirroring the participant hand/limb activity. Optionally,the dashboard may only display a real-time view of the participant'shand/limb activity in response to a condition, such as a hand/limbactivity state change. Optionally, a user interface control may beprovided which enables the user to configure the conditions (e.g., anincrease or decrease of hand/limb activity from a mean hand/limbactivity level of the participant, hand/limb activity greater than athreshold or less than a threshold, etc.) when a view of theparticipant's hand/limb activity may be displayed and for how long.Optionally, a user interface control may be provided which enables theuser to instruct that the display window is to be moved to differentareas of the monitor. Optionally, the display window may be minimized inresponse to a user interface control selection, condition(s), and/ortime period expiration. Optionally, hand/limb activity monitoring audiocues may be presented to a user whether the dashboard display isminimized or not. Optionally, the user interface controls describedherein include but are not limited to one or more of the following: ahand gesture (e.g., a hand/limb gesture detected by a tracking device),a hand selection by the user/operator of a control (e.g., a touch screencontrol, a wireless wand control, a computing device keyboard, etc.), avoice utterance control (e.g., an audio signal such as a whistle, grunt,etc.), a spoken word control (e.g., using a specific voice command suchas “smaller” or “minimize”), a facial gesture (e.g., a wink or twoconsecutive winks, open mouth, etc.), etc. Optionally, audio/voicecontrol commands are optionally muted in response to the user selectingor speaking a mute signal (by speaking the word “mute”), therebypreventing other simulation participants from hearing spoken commandsfrom the user. Optionally, the computing system is configured with aconfigurable audio communication delay enabling the system to detect acontrol shift keyword (e.g., “Mute”) before streaming speech from auser/operator over a communication channel with other simulationparticipants. Optionally, the system provides a whisper mode, whereindetection of whispered commands or spoken audio below a specified ordetermined volume threshold causes the system to prevent such whisperedcommands or spoken audio from being transmitted to other participants ofthe simulation.

Optionally, hand/limb activity monitoring information is presentedsubstantially in real-time to a user/operator in the form of a statusindicator light. For example, if a simulation session participant is ina training session from which enhanced hand/limb activity monitoringdata is being received, a small red indicator light may be presented(e.g., in a portion of the user/operator's display) to a user inresponse to the computing system determining that the subject's (e.g.simulation participant's) hand/limb activity is above a certain range orindicates a certain emotional state, for example, exasperation.Optionally, visual indicator cues may be provided by the computingsystem only in response to detecting certain specified hand/limbactivity variances and/or in the absence of certain hand/limb activityvariances. For example, in the use case above, the computing system maydisplay a continuous visual cue, such as a green light indicator, inresponse to determining that a participant's hand/limb activity mayindicate the participant may be in a relaxed and/or happy state, and ared indicator may be displayed in response to determining that theparticipant's hand/limb activity may be indicative of a tense or angrystate.

With respect to hand/limb activity monitoring audio cues, optionally, anaudio alert may be generated in response to a hand/limb activitymonitoring state change. For example, in reference to the use caseabove, the system may generate a variance audio tone or whisper when thecomputing system determines a participant's hand/limb activity may havechanged from an indication of a participant's relaxed state to a tenseor angry state. Optionally, a periodic (e.g., every 10 sec, 20 sec, 30sec, 1 min, etc.) compliance audio cue/signal is played to theuser/operator as long as the subject's hand/limb activity may indicate,for example, a relaxed or happy state.

As described herein, the computing system optionally monitors hand/limbactivity state changes of a participant and subject to certainconfigured constraints/limitations the user/operator is alerted. Forexample, a user/operator may be alerted in substantially real-time inresponse to determining that the participant's hand/limb activity is nolonger in an expected state. For example, it may be determined that aparticipant hand/limb activity is indicative of an unhappy ordisappointed state during a presentation, the user/operator may bealerted to such state of the participant. Optionally, hand/limb activitydata may be continually compared against a knowledge base which mayenable the computing system to infer certain characteristics/attributesof the participant's emotional state or state-of-mind. For example, in atraining use case, a low rate of hand/limb activity during apresentation may infer the session participant is relaxed and in generalagreement with points being made by the presenter or may be bored.Alternatively, a high rate of hand/limb activity may indicate a sessionparticipant is in disbelief with aspects of the presentation, excited,uneasy, tense, and/or angry. As the foregoing two examples illustrate,the hand/limb activity may be combined with other monitored activity(e.g., facial expressions) to determine the participant's emotionalstate or state-of-mind (e.g., a high hand/limb activity combined withpursed lips may indicate disbelief in certain aspects of apresentation). Optionally, hand/limb activity monitoring may be recordedduring an entire simulation session. Optionally, hand/limb activitymonitoring may only be recorded during certain visual and/or audioevents and/or conditions (e.g., in response to a state change in thesimulation, an introduction of an element into the simulation, inresponse to a change in a participant's hand/limb activity, etc.)

Optionally, enhanced hand/limb activity monitoring data is recorded,stored, and data mined/analyzed subsequent to the simulation session.Optionally, participants are interviewed subsequent to the simulationsession to determine state-of-mind, reactions, attitudes, physiological,and other psychological criteria to create a knowledge base of hand/limbactivity in different contextual virtual reality simulations.Optionally, the hand/limb activity readings may be integrated into adata store to create a knowledge base of hand/limb activity and/orhand/limb activity attributes corresponding to a participant'sstate-of-mind and/or emotion during a simulation or other simulations.This knowledge base may be used to make certain direct inferences fromhand/limb activity monitoring data or indirect inferences when hand/limbactivity monitoring data may be combined with other aspects of thesimulation (e.g., a likelihood of a product or service sale, expectedsocial media feedback, etc.) Optionally, a created knowledge base fromvirtual reality enhanced hand/limb activity monitoring and/or generalresearch related to hand/limb activity monitoring may be used to score asession wherein the score is associated with the likelihood the sessionparticipant may take a certain action (e.g., purchase a good orservice). Optionally, the computing system records/logs relatedinformation associated with or at a time of a participant's hand/limbactivity change of state. In association with a detected participant'shand/limb activity change of state, the system may record some or all ofthe following information: the time and date, a continuous voicesnippet/phrase spoken prior to or during the hand/limb activity changeby the participant or other user/participant in the simulation, othersensory detected attributes associated with the participant (e.g.,expression, heart activity, perspiration, etc. as further describedherein), other body and limb action and/or muscular reactions, etc.Optionally, the computing system converts received/recorded speech intotext enabling improved data mining including search and compare.Optionally, the computing system stores in computer memory recordedspeech and/or converted speech into text in association with auser/operator or participant identifier (e.g., a user name, accountnumber, etc.)

Optionally, certain audio and/or visual information and/or cues may beprovided to a user/operator if a selected participant's detectedhand/limb activity is determined by the system to change from aconfigured state. An optional aspect of the computing system enables auser/operator to designate a hand/limb activity characteristic (e.g.,praying hands) of a participant in which monitored feedback may berecorded and/or provided to the user/operator in substantiallyreal-time. Optionally, a simulation is developed with specific hand/limbactivity states in which tracking may be associated (e.g.,automatically). Optionally, the user/operator is presented with a listof hand/limb activity tracking states/ranges prior to or at the launchof the simulation and may select a certain type/patter of hand/limbactivity and/or amount of hand/limb activity via a user interfacecontrol (e.g., a pull down menu of likely hand/limb activity states).Optionally, the user/operator may remove said selected options via auser interface control of the computing system.

Optionally, each participant (or a subset of participants) in amulti-user, virtual reality simulation session performs an initialcalibration step. The computer system may utilize the calibration todetermine, for example, hand/limb activity rate and/or contentbaselines, from which the user's/participant's/subject's hand/limbactivity monitoring is tracked throughout the simulation session.

FIG. 4 and FIG. 5 illustrate an example process workflow of an examplesystem discussed above. Process states are listed on the left andelements of the operating environment of FIGS. 1, 2, and 3 are listedacross the top. In this example, a multi-user (three-member), virtualreality session occurs in which enhanced tracking is configured for adesignated participant. A trainer is providing instructions to twoparticipants on how to perform a certain repair on a new ACME automobileengine. One of the participants, the trainee, is expected to understandthe repair task in a level of detail that the trainee may be able torepair an ACME engine without assistance. The second participant,assistant, generally provides assistance to the trainee in any repairtask. The trainer, in additional to performing the role of trainer, alsoprovides certain administrative tasks related to launching andconfiguring the VR simulation session. As this example is intended toillustrate certain aspects of enhanced tracking not each and every stepin a conventional simulation initialization and/or simulation session isdescribed.

State 1. An instance of a master server program 205 launches on thecomputing server/device 200 (e.g., in response to a launch commandprovided by the trainer via monitor 260).

State 2. The master server program 205 generates and provides fordisplay a user interface on a console/monitor 260 that identifies thenetwork connected computing devices 210, 211, and 220 (e.g., usingdifferentiating identifiers, such as an alphanumeric name and/or anicon). The master server program 205 receives, via the user interface, aselection by the trainer of an add control, and in response, the masterserver program 205 creates an instance of a client 215 on the computingdevice 210 and binds (e.g., creates a client-server association/linkageenabling the master to send master initiated requests and data over acommunication channel and receive client responses, client initiatedrequests, and data or other computer programming methods of softwareprogram/module association as known to those skilled in the art of datacommunications and/or software development) the created client 215 withthe master server program 205.

State 3. The actions of State 2 are repeated to create and bind client216 on the computing device 211 with the master server program 205.

State 4. The actions of State 2 are repeated to create and bind client225 on the computing device 220 with the master server program 205.

State 5. The master server program 205 initializes the simulation (e.g.,in response to determining that the trainer has selected an initializecontrol presented via a user interface of the master server program 205presented on console/monitor 260), and downloads, over the network 250,to each client computing device's data store 212, 213, 222 executablecode and resources to be used in the simulation. In States 1, 2, 3, or 4(or in a separate state) the tracking cameras 310-340 are alsoinitialized and calibrated (e.g., with respect to any marker trackingdevices used in the simulation including, for example, a PPT eyes usedfor tracking head and body movements during the simulation). Inaddition, any unique infrared marker IDs (e.g., by each marker emittinga distinctive pattern) of the tracking devices are associated withtrainer 150, trainee 155, assistant 160 and clients 215, 216, 225,respectively.

State 6. The simulation session begins (e.g., in response to detectingthat the trainer 150 selected a user interface control of the masterserver program 205 presented in a display (e.g., a head mounted displayor in the monitor 260)). In response to the control selection, themaster server 200 initiates a simulation in each client 215, 216, 225(e.g., via a sync call/command/instruction).

State 7. In response to the VR simulation launch, a display/scene in aRoom is rendered (or caused to be rendered) in the head mounted display170, 173, 175 of each user 150 (trainer), 153 (trainee), 155 (assistant)via the client software 215, 216, 225. In this example, the trainer's150 viewpoint/display includes avatars (e.g., a virtual representation)of the trainee 153 and of the assistant 155. The trainee's 153viewpoint/display includes avatars of the trainer 150 and of theassistant 155. The assistant's 155 viewpoint/display includes avatars ofthe trainer 150 and of the trainee 153.

State 8. The master server program 205 designates the trainee 153 as anenhanced tracking candidate (e.g., in response to determining thetrainer 150 has selected a control presented via a user interface of themaster server program 205 presented on console/monitor 260), causing atleast in part a camera 100 (illustrated in FIG. 1 outside of the headmounted display but in this example embedded within the head mounteddisplay 173 to enable the camera to record the eye/gaze of trainee 155)used for eye/gaze tracking associated with the trainee 153 to be paired(e.g., via a wireless Bluetooth® connection) with the client softwareprogram 216 and hosting computing device 211.

State 9. The master server software program 205 receives analert/notification from the client software program 216 of the eye/gazetracking camera 100 pairing and establishes a virtual data communicationpath between the eye/gaze tracking camera 100 and the master server 200enabling the master server software program 205 to receive trackingcamera 100 output. In this example embodiment, the master serversoftware program 205 receives raw/unprocessed video output. Optionally,the video output is stored in the data store 202. In the followingillustrative states, the eye/gaze tracking camera 100 video outputcontinually received by the master server software program is not shownin the process flows of FIG. 5.

State 10. The master server software program 205 causes the client 215to display an enhanced tracking dashboard 300 in a portion of thetrainer's display 170 (e.g., a head mounted display). The dashboarddisplays the eye/gaze tracking camera 100 view of the trainee's 153eyes. This enables the operator to confirm the camera 100 video outputis being received. Optionally, the tracking camera 100 video output ismirrored to client computing device 211 and the master server computingdevice 220 in order to reduce and/or eliminate display latencies. Thetrainer 150 calibrates the eye/gaze tracking camera 100 and masterserver program 205 by verbally instructing the trainee 153 to directtheir attention to the displayed ACME engine in the trainee's 153 headmounted display 173.

State 11. The master server software program 205 records the eye/gazeposition of the trainee 153 (e.g., in response to detecting that thetrainer 150 selected a user interface control (e.g., by a PPT wandcontrol selection)) subsequent to the verbal instruction of State 10.The position is stored in computer memory and data store 202.

State 12. The master server software program 205 records (e.g., inresponse to determining the trainer 150 having selected a controlpresented via a user interface of the master server program 205presented on console/monitor 260 (e.g., a pull down menu of items ofitems associated with the simulation)) that the trainer 150 is to beprovided an alert monitor indicator 310 (e.g., a green indicator in aportion of the trainer's display 170 (e.g., head mounted display)) whenthe trainee's gaze is determined by the master server program 205 to bedirected at the ACME engine (e.g., a specific object of the simulation).In this example, the alert monitor indicator 310 remains in a portion ofthe screen (e.g., dashboard display) for the duration of the period inwhich the master server program 205 determines the trainee's eye/gaze isdirected towards the ACME engine. (In this example embodiment, the alertmonitor indicator 310 may be continually displayed with certain statusduring the simulation including when the enhanced tracking displaydashboard is minimized. Optionally, the alert monitor indicator 310 doesnot include a border as illustrated in FIG. 3.)

In this example embodiment of enhanced tracking, the computing system(e.g., the master server program 205 and computing device 200) determinethe trainee's gaze is directed at an object of interest (e.g., the ACMEengine) by using received eye position data from the eye/gaze trackingcamera 100 (e.g., direction of gaze of the pupils of the participant inthe X and Y coordinate space) in conjunction with received head/bodyposition tracking data in 6 degrees of freedom as determined fromreceived marker tracking cameras 310-340 video output. For example, ifan object of interest is displayed in the center of the participant'sdisplay as determined by the participant's head/body position and theparticipant's eye/gaze position is similarly directed at the center ofthe participant's display, then the computing system may indicate theparticipant is gazing at the object of interest. In another example, ifan object of interest is displayed in the far right of the participant'sdisplay as determined by the head/body position (e.g., the participantis looking away to the far left of the object) and the participant'seye/gaze position is directed at the center of the participant'sdisplay, then the computing system may indicate the participant isgazing away from and/or not at the object of interest. In anotherexample embodiment of enhanced tracking, the computing system (e.g., themaster server program 205 and computing device 200) may determine theparticipant's gaze is directed at an object of interest by usingreceived eye position data from the eye/gaze tracking camera 100 (e.g.,direction of gaze of the pupils of the participant in the X and Ycoordinate space) and the known (e.g., known by the computing system)displayed position/coordinates of the object of interest on thetrainee's head mounted display 170.

State 13. The master server software program 205 records (e.g., inresponse to determining the trainer 150 has selected a control presentedvia a user interface of the master server program 205 presented onconsole/monitor 260 (e.g., a pull down menu of items associated with thesimulation)) that the trainer 150 is to be provided with an alert (e.g.,a blue indicator in a portion of the trainer's display) when the masterserver determines the trainee's gaze is determined by the master serverprogram 205 to be directed at the trainer (e.g., a participant of thesimulation).

State 14. The master server software program 205 records (e.g., inresponse to determining the trainer 150 has selected a control presentedvia a user interface of the master server program 205 presented onconsole/monitor 260 (e.g., a pull down menu of items associated with thesimulation)) that the trainer 150 is to be alerted (e.g., via analerting tone) if certain real-time eye/gaze analytics executing on themaster server 200 (or other computing server such as computing server230) determine, based in part on the trainee's eye/gaze pattern, thatthe trainee 153 may be confused.

State 15. After the configuration and initialization steps, the trainer150 starts the training session and begins by presenting introductoryremarks. During these introductory remarks, the computing system (e.g.,master server program 205 and computing device 200) determines from theX-Y coordinates of the trainee's eye/gaze from the tracking camera 100and/or head position coordinates as determined in part from the trackingcameras 310-340 output, that the trainee's eye/gaze is directed at thetrainer 150. Based at least in part on the gaze determination, thecomputing system causes the alert indicator 310 to display a blue alert.Optionally, the time of, and period of time in which, the trainee's gazeis directed at the trainer 150 is similarly logged as an enhancedtracking event to the data store 212, 213 and/or data store 202.

State 16. In the course of presenting introductory remarks the trainer150 notices the trainee 153 appears to be directing their gaze towardsher based on a visual observation of the head position of an Avatarcorresponding to the trainee 153. The trainer 150 confirms the trainee153 is gazing towards her by observing the blue alert indicator 310displayed in the enhanced tracking dashboard display.

State 17. The trainer 150 continues the training session afterintroductory remarks by describing certain components of the ACMEengine. During the ACME engine remarks, the computing system (e.g.,master server program 205 and computing device 200) determines from theX-Y coordinates of the trainee's eye/gaze from the tracking camera 100and/or head position coordinates as determined in part from the trackingcameras 310-340 output, that the trainee's eye/gaze is directed at thedisplayed ACME engine. Based at least in part on the gaze determination,the computing system causes the alert indicator 310 to display a greenalert. Optionally, the time of, and period of time in which, thetrainee's gaze is directed at the ACME engine is similarly logged as anenhanced tracking events to the data store 212, 213 and/or data store202.

State 18. During the ACME engine remarks, the trainer 150 notices thetrainee gaze appears to be directed towards the ACME engine by virtue ofa visual observation of the head direction of the Avatar correspondingto the trainee 153. The trainer 150 confirms the trainee 153 is gazingtowards the ACME engine based on an active green indicator displayed inthe enhanced tracing dashboard display.

State 19. During an important step in the training session relating tothe ACME engine repair, the master server 200 determines, for examplevia real-time software analytics corresponding at least in part to theeye/gaze pattern of the trainee 153, that the trainee 153 may becurrently experiencing some confusion. Illustrative examples of eye/gazeanalytic confusion may include, for examples: tracking eye/gaze activitydirected to elements of the ACME engine which may not be relevant to therepair step, tracking eye/gaze directional pattern not expected basedupon a comparison of historic eye/gaze activity for the given repairstep(s), recorded eye movement rates higher than a threshold level,directional eye/gaze tracking directed between the trainer 150, ACMEengine, and/or assistant 155 with a high frequency, etc.

State 20. In response to the determination of potential confusion, themaster server program 205 generates an audio notification to the trainer150 (e.g., a short duration audible tone) and, in this example, causesthe display of the enhanced tracing dashboard (e.g., presented from aminimized state) in a portion of the trainer's display 170 (e.g., headmounted display) via computing device 210 and client 215. The dashboarddisplays a message/notice/alert that the trainee 153 may be currentlyexperiencing confusion. The trainer 150 repeats the training step at aslower pace. Optionally, the time of the confusion alert is logged asone or more enhanced tracking events to the data store 212, 213, and/ordata store 202 along with other information which may have preceded thealert including the training step, audio preceding the alert, eye/gazeposition and activity, etc.

Thus, as described above, an example multi-user virtual realitysimulation may be provided via a client/server architecture withenhanced tracking. Advantageously, the enhanced tracking informs anoperator/designated user/observer of certain attributes and/orcharacteristics of a one or more participant regarding the simulationexperience and/or certain aspects of the simulation. In addition, thesimulation data may be collected over a series of simulations in orderto create a data store of participant activities, attributes, and/orcharacteristics which may be data analyzed in real-time or subsequent toa simulation to determine additional aspects regarding a participant, aparticipant's emotional state, a participant's state-of-mind, and/ortheir simulation experience.

Collaborative Virtual Reality

An aspect of this disclosure relates to systems and methods for enablinga multi-user virtual reality and/or augmented reality experience, suchas, by way of example, those described in U.S. patent application Ser.No. 14/971,720, filed on Dec. 16, 2015 (ATTORNEY DOCKET: WVIZ.003A), thecontents of which are incorporated herein by reference in theirentirety.

SUMMARY

An aspect of the disclosure relates to an example cluster computingsystem configured to interface and work in conjunction with a trackingsystem, to provide a high quality multi-user, virtual and/or augmentedreality experience.

DETAILED DESCRIPTION

An aspect of this disclosure relates generally to devices and methodsfor enabling a multi-user virtual or augmented reality experience, andin particular, to a cluster computing system which works in conjunctionwith a motion capture tracking element.

An aspect of the disclosure relates to an example cluster computingsystem configured to interface and work in conjunction with a trackingsystem, to provide a high quality multi-user, virtual and/or augmentedreality experience, with significantly reduced human-perceptiblelatencies. Optionally, the example cluster computing system furtherenables content providers to create single-user virtual and/or augmentedreality experiences, where the cluster computing system extends thesingle-user virtual and/or augmented reality experience via acooperative distributed and centralized computing architecture to amulti-user virtual and/or augmented reality experience (e.g., by havinga master server generating and enabling the of rendering the differentviewpoints for each client in a multi-user VR simulation the clients.Optionally, the cluster computing system further manages via abstractionthe intricacies of interworking in a multi-user environment with varioushardware components from various hardware manufacturers.

Advantageously, virtual reality and/or simulated environments asdescribed herein can be used in a number of application areas includingbut not limited to: real estate, architecture, product design, humanbehavior analysis, user training, gaming, product and marketingevaluation, design verification, data visualization, teleoperation, andtelepresence or any physical environment that for reasons of, forexample, cost or potential harm to a user can be simulated rather thandirectly experienced. In addition, virtual reality as described hereinmay also be used in augmented reality environments, including forexample, in an automotive augmented reality environment.

Conventionally, VR and AR environments have been oriented to asingle-user environment. Recently, multi-user VR environments/worldshave been created which enable users to collaborate and share a virtualworld experience. A number of technical challenges arise in theimplementation of a multi-user VR environments. Generally, users are onseparate computing platforms connected by a Local-Area-Network or insome cases a Wide-Area-Network (e.g., if the VR participants aregeographical separated) and these networks are subject to inherentcommunication delays, which may result in an unrealistic simulation ofviews and actions. Another key challenge of multi-user VR systems is thetimely synchronization and rendering of a VR participant's action fromthe viewpoint (e.g., correct perspective) of each participant in thesimulation. Further, each VR participant may be configured withdifferent hardware including but not limited to different: computingplatforms, displays, trackers, and input devices. Thus, there is a needto enable a dynamic and real-time (or near real-time) immersive,multi-user virtual reality environment in which user actions and scenesare synchronized across varying hardware devices with minimalprogrammatic and configuration effort on the part of virtual realitycontent creators. Advantageously, embodiments of the virtual/augmentedreality cluster system achieve some or all these objectives via aclient-server architecture which enables certain client contextualprocessing and certain centralized server processing and synchronizationcontrol in order to provide a multi-user virtual user experience whichseems lifelike to all participants of the simulation.

Aspects of the disclosure relates to an example cluster computing systemconfigured to interface and work in conjunction with a tracking system,to provide a high quality multi-user, lifelike, virtual and/or augmentedreality experience, with significantly reduced human-perceptiblelatencies. Thus, aspects of the disclosure herein relate to a multi-userVR/AR environment that may be configured to enable both timely useractions in a display in the first person (that is, the system'srendering of a user action in the viewpoint of the user, optionally, ona computing device local and/or associated with a user) and the view ofa user's action by other participants of the simulation (optionally,rendered on a computing device local and/or associated with a user).Aspects of the disclosure further relate to systems and processes thatprovide timely synchronization (for example, via a master computingdevice) and an enabled rendering of a VR participant's action from theviewpoint (e.g., the correct perspective) of each participant in thesimulation. Further, aspects of the disclosure relate to systems andprocesses that provide a dynamic and real-time (or near real-time)immersive, multi-user virtual reality environment in which user actionsand scenes are synchronized across varying hardware devices, optionallywith minimal programmatic and configuration effort on the part ofvirtual reality content creators/operators/participants.

A given embodiment may include some or all of the features,functionality, systems and methods described herein.

An example embodiment provides a cluster system, comprising: a mastercomputing device; a first computing device configured as a first clientcomputing device, the first client computing device associated with afirst user; a second computing device configured as a second clientcomputing device, the second client computing device associated with asecond user; a plurality of motion tracking cameras, wherein theplurality of motion tracking cameras are configured to reside in aphysical space coincident with the first user, the plurality of motiontracking cameras configured to detect infrared light and to communicateposition data derived from detected infrared light to the firstcomputing device and to the master computing device; a first marker,attachable to a head mounted display associated with the first user, thefirst marker configured to emit a first repetitive pattern of infraredlight pulses, the first marker associated with one or more privilegedoperations that are to be performed locally by the first computingdevice; a second marker, attachable to the head mounted displayassociated with the first user, the second marker configured to emit asecond repetitive pattern of infrared light pulses distinct from thefirst repetitive pattern of infrared light pulses, the second markerassociated with one or more privileged operations that are to beperformed locally by the first computing device; a third marker,wearable or holdable by the first user, the third marker configured toemit a third repetitive pattern of infrared light pulses distinct fromthe first repetitive pattern of infrared light pulses and distinct fromthe second repetitive pattern of infrared light pulses, wherein thethird marker is associated with one or more non-privileged operationsthat are to be performed remotely by the master computing device;non-transitory media that stores instructions readable by the clustersystem, that when executed by the cluster system, cause the clustersystem to: access configuration information comprising informationindicating what types of operations are privileged and what types ofoperations are non-privileged; receive, at the first computing devicefrom one or more of the plurality of motion tracking cameras, positiondata corresponding to the first marker and position data correspondingto the second marker; at least partly in response to the receipt by thefirst computing device of the position data corresponding to the firstmarker and the position data corresponding to the second marker and onthe accessed configuration information, determine that a privilegedoperation is to be performed by the first computing device and not bythe master computing device, the privileged operation enabling anorientation of a first scene of a simulation to be rendered in the headmounted display associated with the first user in synchronization withmovement of the head mounted display in at least five degrees offreedom; transmit, from the first computing device to the mastercomputing device, a position of the first user in the physical space andprovide a first instruction by the master computing device enabling animage corresponding to the first user to be rendered at a first virtualposition in a display device associated with the second user, whereinthe first virtual position is based at least in part on the position ofthe first user in the physical space and on a viewpoint of the seconduser; receive, at the master computing device from one or more of theplurality of motion tracking cameras, position data corresponding to thethird marker; and at least partly in response to receipt of the positiondata corresponding to the third marker and on the accessed configurationinformation, determine that a non-privileged operation is to beperformed by the master computing device and not by the first computingdevice, wherein the non-privileged operation comprises enabling:synchronized movement of a virtual representation of the third markerfrom a viewpoint of the first user corresponding to received positionchanges associated with the third marker to be rendered in the headmounted display associated with the first user and synchronized movementof a virtual representation of the third marker from a viewpoint of thesecond user corresponding to received position changes associated withthe third marker to be rendered in the head mounted display associatedwith the second user; and, optionally wherein the plurality of motiontracking cameras are configured to detect infrared light and tocommunicate position data derived from detected infrared light to thesecond computing device; and, optionally wherein the third marker isattached to a limb of the first user; and, optionally wherein the thirdmarker is attached to or included as a component of a handheld trackingdevice; and, optionally wherein the first repetitive pattern of infraredlight pulses comprise light intensity state transitions; and, optionallywherein the plurality of motion tracking cameras are configured toderive the identity of a marker from a detected repetitive pattern ofinfrared light pulses; and, optionally wherein the cluster system isconfigured to enable, via a user interface of the master computingdevice, a third user to indicate what types of operations are privilegedand what types of operations are non-privileged; and, optionally whereinthe virtual marker representation of the third marker comprises avirtual representation of a glove.

An example embodiment provides a cluster system, comprising: a mastercomputing device; a plurality of motion tracking cameras, wherein theplurality of motion tracking cameras configured to reside in a physicalspace coincident with the first user, the plurality of motion trackingcameras configured to detect infrared light and to communicate positiondata derived from detected infrared light to the master computingdevice; a marker, wearable or holdable by the first user, configured toemit a repetitive pattern of infrared light pulses, wherein the markeris associated with one or more non-privileged operations that are to beperformed remotely by the master computing device; non-transitory mediastoring instructions readable by the cluster system, that when executedby the cluster system, cause the cluster system to: access configurationinformation comprising information indicating what types of operationsare privileged and what types of operations are non-privileged; receiveat the master computing device, a position of the first user in thephysical space and provide a first instruction by the master computingdevice enabling an image corresponding to the first user to be renderedat a first virtual position in a display device associated with a seconduser, wherein the first virtual position is based at least in part onthe position of the first user in the physical space and on a viewpointof the second user; receive, at the master computing device, from one ormore of the plurality of motion tracking cameras, position datacorresponding to the marker; and at least partly in response to thereceipt of the position of the marker and on the accessed configurationinformation, determine that a non-privileged operation is to beperformed by the master computing device and not by another computingdevice associated with the cluster system, wherein the non-privilegedoperation comprises enabling: synchronized movement of a virtualrepresentation of the marker from a viewpoint of the first usercorresponding to received position changes associated with the marker tobe rendered in the head mounted display associated with the first userand synchronized movement of the virtual representation of the markerfrom a viewpoint of the second user corresponding to received positionchanges associated with the third marker to be rendered in the headmounted display associated with the second user; and, optionally whereinthe marker is configured to be attached to a limb of the first user;and, optionally wherein the marker is attached to or included as acomponent of a handheld tracking device; and, optionally wherein thefirst repetitive pattern of infrared light pulses comprise lightintensity state transitions; and, optionally wherein the plurality ofmotion tracking cameras are configured to derive the identity of amarker from a detected repetitive pattern of infrared light pulses; and,optionally wherein the cluster system is configured to enable, via athird user interface of the master computing device, a user to indicatewhat types of operations are privileged and what types of operations arenon-privileged.

An example embodiment provides a cluster system, comprising: a firstcomputing device configured as a first client computing device, thefirst client computing device associated with a first user; a secondcomputing device configured as a second client computing device, thesecond client computing device associated with a second user; aplurality of motion tracking cameras, wherein the plurality of motiontracking cameras are configured to reside in a physical space coincidentwith the first user, the plurality of motion tracking cameras configuredto detect infrared light and to communicate position data derived fromdetected infrared light to the first computing device; a first marker,attachable to a head mounted display associated with the first user, thefirst marker configured to emit a first repetitive pattern of infraredlight pulses, the first marker associated with one or more privilegedoperations that are to be performed locally by the first computingdevice; a second marker, attachable to the head mounted displayassociated with the first user, the second marker configured to emit asecond repetitive pattern of infrared light pulses distinct from thefirst repetitive pattern of infrared light pulses, the second markerassociated with one or more privileged operations that are to beperformed locally by the first computing device; a third marker,wearable or holdable by the first user, the third marker configured toemit a third repetitive pattern of infrared light pulses distinct fromthe first repetitive pattern of infrared light pulses and distinct fromthe second repetitive pattern of infrared light pulses, wherein thethird marker is associated with one or more non-privileged operationsthat are to be performed remotely by a computing device other than thefirst computing device; non-transitory media that stores instructionsreadable by the cluster system, that when executed by the clustersystem, cause the cluster system to: access configuration informationcomprising information indicating what types of operations areprivileged and what types of operations are non-privileged; receive, atthe first computing device from one or more of the plurality of motiontracking cameras, position data corresponding to the first marker andposition data corresponding to the second marker; at least partly inresponse to the receipt by the first computing device of the positiondata corresponding to the first marker and the position datacorresponding to the second marker and on the accessed configurationinformation, determine that a privileged operation is to be performedlocally by the first computing device and not by a computing deviceother than the first computing device, the privileged operation enablingan orientation of a first scene of a simulation to be rendered in thehead mounted display associated with the first user in synchronizationwith movement of the head mounted display in at least five degrees offreedom; transmit, from the first computing device to a master computingdevice, a position of the first user in the physical space; receive, atthe second computing device, a first instruction enabling: an imagecorresponding to the first user to be rendered at a first virtualposition in a display device associated with the second user, whereinfirst virtual position is based at least in part on the position of thefirst user in the physical space and on a viewpoint of the second user;receive, at the first computing device, a first non-privilegedinstruction enabling: synchronized movement of a virtual representationof the third marker from a viewpoint of the first user corresponding toreceived position changes associated with the third marker to berendered in the head mounted display associated with the first user; andreceive, at the second computing device, a second non-privilegedinstruction enabling: synchronized movement of a virtual representationof the third marker from a viewpoint of the second user corresponding toreceived position changes associated with the third marker to berendered in the head mounted display associated with the second user;and, optionally wherein the plurality of motion tracking cameras areconfigured to detect infrared light and to communicate position dataderived from detected infrared light to the second computing device;and, optionally wherein the plurality of motion tracking cameras areconfigured to detect infrared light and to communicate position dataderived from detected infrared light to a computing device other thanthe first computing device or the second computing device; and,optionally wherein the third marker is configured to be attached to alimb of the first user; and, optionally wherein the third marker isattached to or included as a component of a handheld tracking device;and, optionally wherein the at least five degrees of freedom compriseX-axis, Y-axis, Z-axis, yaw, and roll; and, optionally wherein the firstrepetitive pattern of infrared light pulses comprise light intensitystate transitions; and, optionally wherein the plurality of motiontracking cameras are configured to derive the identity of a marker froma detected repetitive pattern of infrared light pulses.

Example features will now be described with reference to the figures.

FIG. 7 illustrates an example operating environment of a clusteredsystem with a master server and multiple clients. As described herein,the illustrated system architecture can be used to facilitate a virtualand/or augmented reality experience, and in particular, a multi-uservirtual and/or augmented reality experience. The system may beconfigured to perform some or all of the following: image capture andtracking functions, video/image processing, marker identification (e.g.,LED marker identification), user/object position calculations, actionsynchronization, and/or dynamic rendering of virtual worlds in anoptimized (e.g., in substantially real-time and/or minimal renderingdelays) manner to create a lifelike experience. For clarity, thedescription herein may refer to virtual reality, which is intended toencompass both virtual reality and augmented reality, unless the contextindicates otherwise.

As illustrated in FIG. 7, a master server computing device 5200 (e.g., ageneral purpose computer, a tablet computer, a cloud-based server, asmart phone, a graphics processing unit, etc.) is coupled to a pluralityof client computing devices 5210, 5220 over a data network 5250 (e.g., alocal area network, a wide-area network, the Internet, a privatenetwork, a public network, etc.). Optionally, the master server 5200 maybe directly connected to a console/monitor 5260 (or over the datanetwork 5250 to a console/monitor 5260) which displays a user interfacevia master server software 5200 for a user/participant or operator toprovision, designate, and/or configure the master server 5200 (e.g.,download and/or upgrade software, provision data communicationinterfaces, configure accounts, administer security, edit files, etc.).Optionally, the master server 5200 also provides a computing platformfor compiling and/or executing certain programs and/or programmaticscripts (e.g., simulations), receiving and sending certaincommunications, performing video/image processing tasks, performing mathcomputations, displaying/rendering certain images (e.g., renderingvirtual worlds), providing client synchronization, hosting a clientapplication, etc.

A given computing device 5200, 5210, and 5220 may be further configuredwith a data store 5202, 5212, and 5222, respectively, enabling theserver to store in memory data associated with the VR session,simulations, models, objects, images, certain parameters, executablescripts/code, local event logs, error logs, etc. Thus, for example, themaster server computing device 5200 may host a master server softwareprogram 5205, as illustrated in FIG. 11, comprising a single softwareprogram or a plurality of software programs or software modulesincluding, for example, a render engine 5610 configured to render and/orenable the rendering of VR scenes, a physics engine 5615 (e.g., thatprovides a simulation of physical systems, such as rigid and soft bodydynamics, collision detection, and fluid dynamics, and that provides aninterface that hides the low-level details of the physics needed invirtual reality applications to enable application/game developers tofocus on the higher-level functionalities of the application), a rulesengine 5620, a simulation control engine 5625 (that coordinatessimulation execution), a session manager 5630, a simulation statesynchronizer engine 5635 (that, for example, synchronizes associatedclient viewpoints) and/or an error handling 5640, a client-servercommunications manager 5650 (that, for example, manages client servercommunications including over a data communication network (e.g., a lowlatency data communication network)), resource manager 5655 (that, forexample, manages resources, including shared resources (e.g., simulationobjects, scenes, etc.), virtual reality tracking and markeridentification software 5660 (e.g., the Vizard VR™ toolkit and PPTStudio software from WorldViz LLC of Santa Barbara) by way of example.

The master server computing device 5200 may include cluster nodes (e.g.,companion computing devices) that handle gameplay/simulation logic,tracking software, and one or more additional servers that processcommunications from client computing device. The master server computingdevice 5200 may include a login server, including a client/user identityand access authentication engine, that manages login sessions withclient computing devices, validates client computing devices (e.g., bychecking password and device identifiers), and if the client computingdevices are validated, enables the devices to access the mastercomputing device 5200 and/or cluster nodes. The data store 5202associated with the master server computing device 5200 may store useraccount information such as password information, user/deviceidentifiers, status of game/simulation play which may be stored upon auser suspending a game/simulation or other virtual/augmented realityactivity and which may be accessed and utilized when the activity isrestarted to provide a continuous experience, user usage information(e.g., time and duration of virtual/augmented reality activity), otherusers the user may be linked to (e.g., other users the user hasinteracted with when engaging in virtual/augmented reality activities),etc.

The master server computing device 5200 may further comprise multipleservers distributed over a broad geographic area to reduce latency withrespect to interactions with client server devices, wherein a givenserver (or servers) may serve client computing devices in itsgeographical area. The various master server computing devices mayinclude network interfaces and may be interconnected via high speed datanetworks to further reduce latencies. The cluster nodes may includededicated high speed graphic processors to process virtual/augmentedreality graphic tasks. The master server computing device 5200 mayinclude one or more administrator terminals.

Similarly, as illustrated in FIG. 7, one or more client computingdevices 5210, 5220 (e.g., a general purpose computer, a tablet computer,a cloud-based server, a smart phone, a graphics processing unit, a gameconsole, etc.) are coupled to the master server 5200 over a data network5250 (e.g., a local area network, a wide-area network, the Internet, aprivate network, a public network, etc.). Optionally, each client device5210, 5220 can connect with other client devices via the data network5250 (e.g., in a peer-to-peer manner). Optionally, a given clientcomputing device 5210, 5220 may be connected directly or over a networkto a console which displays a user interface via client software 5215,5225 to a user/participant or operator and which can be used toprovision, designate, and configure the given client computing device5210, 5200 (e.g., download and/or upgrade software, provision datacommunication interfaces, configure accounts, administer security, editfiles, etc.). Optionally, a given client computing device 5210, 5200provides a computing platform for compiling and/or executing certainprograms and/or programmatic scripts, receiving and sending certaincommunications, performing video/image processing tasks, identifyingdistinct markers from recorded infrared light, calculating markerpositions in space, performing math computations, enabling thedisplaying/rendering of certain images (e.g., rendering virtual worldsor augmented images), etc. For example, a given client computing device5210, 5200 may include a 3D rendering engine, a 3D sound engine, avirtual reality execution engine, an avatar engine, a simulation statesynchronizer engine, a session manager engine and/or a networkinterface.

FIG. 7 also illustrates an optional element of the cluster system, aplurality of motion capture cameras 5310, 5320, 5330, and 5340. Motiontracking is a crucial component of most virtual reality systems. Bytracking the position and orientation of the user, the cluster systemcan control the simulation's viewpoint such that the user can navigatethrough the virtual world just as naturally as they would navigatethrough the physical world. In addition, tracked movements of anindividual in the physical world may be mapped onto certain movements ofan avatar in the virtual world. Optionally, these motion capture cameras5310-5340 are connected physically via cabling or wirelessly to the datanetwork 5250 and/or directly to the master server 5200, client 5210,and/or client 5200. Optionally, the video output and/or detected markeridentities and marker positions are streamed from the cameras 5310-5340to the master server computing device 5200 and/or the client computingdevices 5210,5200 as further described herein below. Optionally, aplurality of motion capture cameras are positioned above and/or residewithin a staging area in which VR session users participate in asimulation. FIG. 8 illustrates an example camera positioning in asimulation. By way of example, multiple (e.g., 2, 4, 6, 8, 9, 10, 11,12) tracking cameras (e.g., precision position tracking (PPT) cameras,such as the PPT X™ precision motion tracking cameras from WorldViz LLCof Santa Barbara) may stream live video to a master server computingdevice 5200, or a client computing device 5210, 5200, which areprovisioned with virtual reality tracking software (e.g., the Vizard VR™toolkit or PPT Studio from WorldViz LLC of Santa Barbara; dll forWindows; C source library for Linux) capable of image processing livevideo images in substantially real-time.

Optionally in addition or instead, a dedicated computing device 5230(e.g., a laptop, PC, smartphone, server, etc.) may be optionallyprovisioned with the virtual reality tracking software (e.g., Vizard VRand/or PPT Studio from WorldViz LLC of Santa Barbara) capable ofprocessing the received live camera output stream in substantiallyreal-time. Optionally, the motion capture cameras record image dataincluding emitted infrared light in a tracking area. Optionally, therecorded image data is provided (e.g., streamed) to the computing device5230 or other computing device via a data network (e.g., 5250).Optionally, the computing device receiving the image data has certainsoftware (e.g., PPT Studio software from WorldViz LLC of Santa Barbara)capable of determining from the image data, a position and identity foreach infrared marker in the field of view/tracking area of the motioncapture cameras.

Optionally, the calculated/determined/derived marker position andidentity information (e.g., tracking data) may be used (e.g., by thecomputing device) to render a change in a scene or orientation of ascene as viewed by a user (e.g., in a head mounted display or otherdisplay). For example, infrared markers may comprise light emittingdevices (LED) which may be configured to be attached to or mounted on anitem worn by the user (e.g., headgear). Optionally, a marker device canbe attached to articles of clothing and/or strapped or otherwiseattached to or physically associated with a user. Optionally, aninfrared marker tracking device includes two or more light emittingcomponents attached to one or more items worn by the user and the systemtracks the light source in three dimensions to determine the user'smovements in time. Optionally, head movement tracking data is collectedin substantially real-time and can be used, for example, in a virtualreality display in which some or all of the displayed images aremodified coincident with the tracked movement of the user. For example,if the tracking system detects sideways movement of the user, theperspective and/or the displayed image viewed by the user may bemodified to simulate the sideways movement of the user (e.g., displayingan image behind a tree which was obstructing the user's view prior tothe user's sideways movement).

For example, the virtual reality tracking software 5660 may incorporatea visual tool for configuring devices that the VR tracking softwaresupports, including displays (e.g., head-mounted displays, multi-screenprojection walls, consumer 3D monitors), trackers (head trackers,gloves, full body motion capture), and input devices (e.g., wands,steering wheels, gamepads, joysticks, etc.). The VR tracking softwaremay enable editing of transform nodes (e.g., position, rotation, scale),clone/copy nodes, rename nodes, delete nodes, insert new group/transformnodes, and add/remove/modify descriptors. The VR tracking software mayenable the layout of a virtual scene by combining and arranging multipleindependent 3D models. The VR tracking software may enable a givenvirtual reality scene or objects to react to a user's natural bodymotions. Thus, for example, the VR tracking software may enable a userto interact with head, hands, and feet with virtual objects in a mannersimilar to real objects to provide a convincing sensation oftelepresence. The VR tracking software optionally provides full bodymotion capture technology that delivers full human presence for bothfirst person point of view and third person points of view experiences.The VR tracking software optionally enables the integration of live fullfeature avatars.

Other optional features of the PPT X cameras of the clustered system mayinclude some or all of the following: charge-couple device camerasensors; three degrees-of-freedom positioning; six degrees-of-freedom(X, Y, Z, yaw, pitch, roll) positioning with an optional sensor;approximately 5 meter, 15 meter, 30 meter camera range and a recommendedtracking space of approximately 10 meters by 10 meters by 10 meters, orother tracking space; multiple (e.g., 2, 4, 6, 8, 9, 10, 11) independentthree degrees-of-freedom tracking targets (where, for a marker to betracked, preferably the marker is visible by two or more cameras);tracking precision of approximately less than 0.5, 1, 2, or 5millimeters over a 3 meter by 3 meter by 3 meter volume; optical sensorprecision may be about 1:10,000, 1:20,000, 1:30,000 arc at 75 percent;tracking accuracy of approximately less than a 0.1, 0.5, or 1 centimeterover a 3 meter by 3 meter by 3 meter volume; a field of view of 60, 68,75, or 80 degrees horizontal (standard) and 80, 88, 95 degreeshorizontal (wide) for each camera (and not, for example, the resultingtrackable space of the system of cameras); less than 30 seconds, oneminute, or two minutes calibration using a digital calibration rig; a 30HZ, 60 Hz, or 120 Hz update rate; a centralized computing device forimage processing; a maximum cable length to cameras of approximately 15meter, 30 meters 50, meters, or 75 meters; analog camera interface;minimum latency of 10, 18, or 30 milliseconds; interface options (e.g.,RS-232, USB, Bluetooth, WiFi, etc.), 115.2 kilobits per second, streamedor polled and Ethernet; and, ambient conditions utilizing indoorfluorescent or other lighting; camera size of 160 by 70 by 60millimeters and weighting 500 grams. It is understood that dimensions,ranges, and components, other than those recited above, may be used.

Optionally, high performance precision position tracking (PPT) cameras(e.g., 1-32 high performance Precision Position Tracking cameras, suchas the PPT E™ precision motion tracking cameras from WorldViz LLC ofSanta Barbara) configured to record image data, detect infrared light,and determine the identity and position of detected infrared markers andoutput stream two dimensional centroid coordinates of the detectedinfrared markers in the camera field of view (e.g., as similarlydescribed above for the PPT X but via onboard camera image/videoprocessing in the case of PPT E), among other features, to the masterserver computing device 5200 and/or the client computing device 5210,5200. Optionally, the marker position data (e.g., tracking data) may beused (e.g., by the computing device) to render a change in a scene ororientation of a scene as viewed by a user (e.g., in a head mounteddisplay or other visible display). Other features of the PPT cameras ofthe clustered system may include some or all of the following: color orblack and white complementary metal-oxide-semiconductor camera sensors;three degrees-of-freedom positioning; six degrees-of-freedom (X, Y, Z,yaw, pitch, roll) positioning with an optional sensor; approximately 5,10, or 20 meter camera range or other working camera range, and arecommended tracking space of approximately 20 meters by 20 meters by 10meters, or other tracking space (where, for a marker to be tracked,preferably the marker is visible by two or more cameras). Overalltracking space dimensions may be determined by the camera range, cameralayout, and background lighting. Other optional features may include1-32 independent three degrees-of-freedom tracking targets; trackingprecision of approximately less than one-quarter of a millimeter over a3 meter by 3 meter by 3 meter volume; optical sensor precision is1:20,000 arc at 75 percent; tracking accuracy of approximately less thana quarter of a centimeter over a 3 meter by 3 meter by 3 meter volume; afield of view for a given camera of 79 degrees horizontal (standard) and98 degrees horizontal (wide) for each camera (and not, for example, theresulting trackable space of the system of cameras); less than oneminute calibration using a digital calibration rig; 180 Hz update rate;centralized computing device for image processing; maximum cable lengthto cameras of approximately 200 meters; analog camera interface; minimumlatency of approximately 20 milliseconds; interface options includeEthernet, using VRPN (Virtual-Reality Peripheral Network); compatibilitywith certain ambient conditions (e.g., indoor fluorescent lighting); andcamera size of 45 by 32 by 92 millimeters and weighting 145 grams. It isunderstood that dimensions, ranges, and components other than thoserecited above, may be used, such as those discussed with respect toother embodiments.

As previously described herein, the motion tracking cameras 5310-5340record emitted infrared light pulses (e.g., emitted infrared lightpulses) from tracking emitter devices associated with a VR participant.An example tracking device which emits trackable infrared lightsignatures is a wireless Precision Position Tracking (PPT) device, PPTEyes™, from WorldViz LLC of Santa Barbara. The PPT Eyes is a motiontracker mountable to 3D glasses or VR head mounted displays whichprovides wide-area tracking and can be integrated with or attached todisplay headgear enabling the tracking of a user's head and/or bodymovements when combined with virtual reality tracking software. Thetracking software, which optionally executes on board the camera5310-5340, on the master server 5200, a standalone computingdevice/server 5230, and/or client computing device 5210, 5200, iscapable of identifying via the streaming camera video the one or moremarkers of the PPT devices.

Other features of a head mountable wireless PPT device may include someor all of the following features: a power button on top (or elsewhere)for easy user access; a housing imprinted with LED identifiers (e.g.,LED1:1 and LED2:R) for clarity with signal identifier codes; arechargeable (e.g., lithium ion) battery with an optional endurance of4, 6, or 8 hours (with low intensity setting) with a range of more than20 meters (with high intensity setting) without impacting performance;optionally factory or self-mounted onto commercially available passiveand active 3D glasses; automatic “go-to-sleep” mode if not moved for afirst period of time (e.g., 30 seconds), and turn off after a secondperiod of times (e.g., 10 minutes) to save on battery power; userswitchable between USB charge mode and USB power mode; easilyreprogrammable device settings (sleep mode and turn-off timer periods,LED Identifiers, LED light intensity) via a simple application (or viasoftware toolkit from WorldViz LLC of Santa Barbara) wherein the deviceis connected to a user's computer via a wired or wireless interface(e.g., a micro USB port); compatible with a plurality of protocols(e.g., TrackD, VRPN, WorldViz PPT Studio™, etc.); six degrees-of-freedom(X, Y, Z, yaw, pitch, roll) enabling the full determination ofleft/right eye locations needed for stereoscopic viewing; angular rangeof a full 360 degrees—all axis; tracking precision of approximately lessthan a quarter of a millimeter over a 3 meter by 3 meter by 3 metervolume with a rotation precision of 0.09 degrees; position accuracy ofapproximately less than a quarter of a millimeter over a 3 meter by 3meter by 3 meter volume with a rotation accuracy of one degree RMS yaw,one quarter of a degree RMS roll; update rate of 180 Hz); latency of nomore than 20 milliseconds; light weight 30 grams (with two infrared reddiffusion balls); device size of 203 by 14 by 32 millimeters; active LEDmode (PPT Marker IDs). It is understood that dimensions, ranges, andcomponents, other than those recited above, may be used, such as thosediscussed with respect to other embodiments.

Optionally, the motion tracking cameras record emitted infrared lightpulses from tracking devices held by a user. An example handheldtracking device which emits trackable infrared light signatures may be awireless Precision Position Tracking (PPT) Wand™ from WorldViz LLC ofSanta Barbara. The PPT Wand is a wireless, battery powered wand. Thewand provides interaction and navigation in immersive virtual realitysystems and may include a joystick/pad and trigger enabling handtracking and hand grabbing and/or moving of virtual objects. Otheroptional features of a handheld wireless tracking device may includesome or all of the following: a rechargeable (e.g., lithium ion) batteryendurance of 8 hours with a range of more than 20 meters withoutimpacting performance; an ergonomic one-handed design, vibrant buttontouch, and a multitude of input modes; automatic “go-to-sleep” mode ifnot moved for one minute, and turn off after 10 minutes to save onbattery power; user switchable between USB charge mode and USB powermode; easily reprogrammable wand settings (sleep mode and turn-off timerperiods, LED Identifiers, LED light intensity) via a simple application(or via software toolkit from WorldViz LLC of Santa Barbara) wherein thewand is connected to a user's computer (e.g., wirelessly, via a microUSB port, or via other interface); compatible with a plurality ofprotocols including TrackD, VRPN, and WorldViz native PPT protocol; sixdegrees-of-freedom (X, Y, Z, yaw, pitch, roll); angular range of a full360 degrees—all axis; tracking precision of approximately less than aquarter of a millimeter over a 3 meter by 3 meter by 3 meter volume witha rotation precision of 0.03 degrees; position accuracy of approximatelyless than a quarter of a millimeter over a 3 meter by 3 meter by 3 metervolume with a rotation accuracy of one degree RMS yaw, one quarter of adegree RMS in pitch and roll; update rate of 180 Hz with PPT E series);latency of 20 milliseconds with PPT E series; light weight 196 grams;wand size of 239 by 65 by 110 millimeters; passive and active LED mode(PPT Marker IDs). It is understood that dimensions, ranges, andcomponents, other than those recited above, may be used, such as thosediscussed with respect to other embodiments.

Other optional motion tracking features which can be employed in theclustered system include but are not limited to: inertial-based motiontracking, magnetic-based motion tracking, optical motion tracking,motion capture suits, and/or other motion tracking systems that may beutilized in virtual reality applications.

In the example clustered system, certain input devices may be providedfor a user to manipulate (e.g., by one or more of the following:grasping, placing an object, finger flexion, abduction between fingers,thumb crossover, palm arch, wrist flexion, wrist abduction, etc.) theenvironment during a virtual reality session. For example, the inputdevice may include one or more of the following examples: gamepads,joysticks, mouse, keyboard, haptic glove, wand, and/or other inputdevices used to manipulate an environment that may be utilized in avirtual reality application.

In the example clustered system, certain displays may be employed topresent a virtual reality experience to a user including but not limitedto: stereophonic displays (e.g., Oculus), a Cave Automatic VirtualReality (CAVE) 3-dimension projector system with single display wall, a2-dimension monitor, a 3-dimension monitor, and/or other display devicesthat may be utilized in a virtual reality application. Optionally, thecluster system integrates motion tracking to enable certain displays,including 2-dimensional displays, to render the worlds appropriately sothat users feel as though they're looking beyond the wall into thevirtual world (e.g., the images adapt to movement so that users feelmore like they are looking through a window than looking at a picture).

Example Multi-User Embodiment

An example of a collaborative Virtual Reality (VR) session illustratescertain optional features and advantages of the system in which useractions and display of such actions are synchronized across varyinghardware devices with minimal programmatic and configuration effort onthe part of VR developers/content creators. Advantageously, because theexample clustered system manages the underlying communications, thesimulation execution and control, and the Avatar positional changes foreach participant in a multi-user simulation, the system enables VRcreators to focus their efforts on the content of a simplifiedsimulation scenario/environment (including for example, a single-usersimulation). Further, this example embodiment illustrates a novelapproach to a multi-user simulation environment which enables localcontextual computing in association with each VR participant in order topresent certain aspects, for example participant's actions of a firsttype, of the simulation in substantially-real time (e.g., with minimaldisplay delays and/or communication latencies, such as those that wouldnot be perceived by a user) to the VR participant. By contrast, certainother aspects of the simulation, for example participant's actions of asecond type, are only partially processed in a local context or are notprocessed at all in the local context but rather are processed in aremote context. Tracking data associated with the second type ofparticipant's action may be transmitted to and received at a masterserver for centralized processing (e.g., at a master server) and thenscene rendering content changes are performed by the master server anddistributed, over a communication medium (e.g., a data network), to eachclient participant in the multi-user session. Therefore, in certainsimulations, the data sent between the master and clients is primarilylimited to the position and orientation of the participants of thesimulation.

In this illustrative example, and with reference to FIG. 6, two usersexperience an interactive VR session (hereinafter referred to as asimulation or VR simulation) in which the users begin in a first roomand one user walks into a second room (it is understood that thisexample assumes two users for clarity, although other scenarios may havemore than two users). FIG. 6 illustrates a configuration of the tworooms 5100, 5105 of the simulation. Room 5100 is configured with asliding door 5110 which enables a user to walk from Room 5100 to Room5105 upon opening the sliding door 5110. In this example, the slidingdoor 5110 and the wall separating Room 5100 and Room 5105 are merely asimulation as there is no physical wall separating Room 5100 and Room5105 in the real world (although Rooms 5100 and 5105 may be physical,real world rooms). Further in this example, Room 5105 is configured withthe adjoining sliding door 5110 and a physical chair facing the door5110. In addition, in this example, the sliding door 5110 slides opensin response to the application of a left-to-right hand movement from auser and automatically closes from right-to-left when no pressure isapplied to the door or door handle. In this example, a first user 5150is initially positioned to the left of a second user 5155 and,correspondingly, the second user 5155 is positioned to the right of thefirst user 5150. Both users begin the VR session/simulation at theeastern portion of the room, each facing west towards the sliding door5110.

In this example, a system of four tracking cameras 5310-5340 (e.g., amotion capture system including, for example, a PPT X camera systemrunning at 60 Hz or a PPT E camera systems running at 180 HZ, both fromWorldViz, LLC of Santa Barbara, Calif.) are configured in the VR sessionin order to track certain user movements of the first 5150 and second5155 users via a set of trackable infrared LED markers attached to orconfigured in association with each user, see also FIG. 8. For example,a precision positioning tracking device, such as the PPT Eyes trackingdevice from WorldViz, may be affixed to a wearable head mounted display5170, 5175 of each user to track each user's head and bodymovements/travel. In this example, each user's right hand is configuredwith a haptic glove 5160 and 5165 respectively, where the haptic gloveis configured to simulate a touch sensation when a user interacts withan object in the simulation to enable a wearer to “feel” a virtualobject as if it really existed, and wherein the glove further includes aone or more trackable infrared LEDs and/or finger movement sensors.

In this example, at least four separate computing devices and/orcomputing servers 5200, 5210, 5200, and 5230 are employed in thesimulation as previously described herein, see also FIG. 7. In thisexample, computing server 5200 is configured as a master computingserver and is provisioned in association with server program software5205 (e.g., WorldViz cluster master from WorldViz LLC of Santa Barbara).In this example, computing device 5210 (e.g., a Graphic Processing Unit(GPU), a processing unit to provide rapid visual/display processing) isprovisioned in association with a software client 5215 (e.g., a WorldVizcluster client from WorldViz LLC of Santa Barbara) and wherein theclient is configured in a client-server relationship with master serversoftware 5205, as further described herein below. Similarly, in thisexample, computing device 5200 (e.g., a GPU) is provisioned inassociation with a software client 5225 (e.g., a WorldViz cluster clientfrom WorldViz of Santa Barbara) and wherein the client is configured ina client-server relationship with master server software 5205, asfurther described herein below. Each of the three computing devices areinterconnected via a local-area or wide-area data communication network5250 as further described above.

In the example embodiment, the cluster system performs a set ofinitialization and/or connection acts, such as a simulation launch,followed by the actual simulation. These initialization and launch acts,and the simulation of the example embodiment are further describedbelow.

In this example, the master server 5205 and clients 5215, 5225initialization precedes the client-server connection process.Optionally, the connection process precedes the initialization process.Optionally, the initialization process may be manually performed by auser (as opposed to being automatically performed by the system) andoccurs independent from any client-server binding/association.

Virtual Reality Initialization

As previously mentioned herein, clustering environments enable thedistribution of computational tasks across multiple computing resourceswithin a network of computers, thus, enabling the simulation ofcomputationally expensive virtual/augmented reality worlds. To avoidbogging down the network (e.g., creating network latencies) during asimulation the amount of data that each computer transmits and receivesis minimized. This is accomplished by, for example, loading the virtualworld on each computing device and then pass only the data that isunique to the local/client computer. In this example, the master server5200 and each client 5210, 5200 are initially seeded with a set ofidentical copies of executable programmatic code and associated data(e.g., resource files and modules), which is loaded during theinitialization process or during runtime. This limits the primarycommunications between the master server 5205 and clients 5215, 5225during the VR session to synchronization calls which keep the viewpointand scenegraph states identical across all clients in the multi-user VRsimulation. The synchronization may be performed in substantially realtime (so that any delay is not perceptible or is barely perceptible, tousers). Optionally, resources, such as models, textures, and audio filesare not generally transmitted from the master to clients during the VRsession.

Optionally, the resources needed for the simulation in this example aredownloaded from the master server's data store 5202 to the clients andstored in each client's data store 5212, 5222, wherein the data storescomprise computer memory and can be configured as a general purpose filesystem, tables, flat databases and/or SQL databases by way ofnon-limiting illustrative example. Optionally, to avoiddistributing/downloading/uploading files and/or copies of files acrosscomputing servers and computing devices, a file sharing network with acommon data store may be employed with absolute, full network paths(e.g., beginning with a drive letter followed by the rest of the path)or relative paths (a path relative to the current directory) to thestored resources specified at each participating computing server and/orcomputing device.

Optionally, the simulation initialization is initiated in response to auser/administrator selection (e.g., the first user 5150, the second user5155, or a third user) of a control at the master server 5205. Themaster server 5205 downloads to each client computing device 5210, 5200the executable code and resources to be used in the VR simulation. Uponreceipt of the executable code and resources, each networked client5215, 5225 loads their respective data store 5212, 5222. Optionally, thecode and resources are downloaded directly by the master into the datastore 5212, 5222). Optionally, any problems and/or detected errors inloading the code and resources are reported to the administrator/userwho initiated the download such that the administrator/user can takecorrective action. Optionally, each client in the network of clients(e.g., 5215, 5225) independently requests (e.g., via a user interfacecontrol of the software clients 5215, 5225) the master server 5205 toinitialize each client's respective data store 5212, 5222. Optionally,any or certain detected errors in client initialization are displayed tothe associated client administrator/user and/or the administrator/userof the master server 5205 in order to take corrective action. Wheninitialization completes (e.g., without fatal errors), the clients 5215,5225 are data store synchronized with the master server 5205 and theinitialization step is complete.

Cluster Connection Management

The cluster client software program enables a client computing device toreceive signals/requests/messages to launch a simulation (e.g., toreceive a signal from an associated/bound master server softwareprogram) and synchronize user actions during a launched simulation(e.g., via a synchronous (sync) update from an associated/bound masterserver program). In this example, the users 5150, 5155 launch theirrespective client software programs 5215, 5225.

The master server software program enables a master computingdevice/server to send signals/requests/messages to launch simulations(e.g., to send a signal to one or more associated/bound clients) andsynchronize user actions across the associated/bound client programs. Inthis example embodiment, a user (e.g., users 5150, 5155) or an operatorlaunches an instance of the master server software program 5205 on thecomputing server 5200. Optionally, the master server program 5205displays, via a user interface, the computing devices 5210, 5200 thatare connected over a communication medium (e.g., a local-area orwide-area data network) and that are running the cluster client softwareprograms 5215, 5225. Optionally, the master server program 5205 userinterface displays the IP address of the data connected computingdevices/servers and the name of each client 5215, 5225. Optionally, theserver program 5205 provides an add control (e.g., a check box control)by which a user 5150, 5155 or an operator of the cluster computingsystem can associate/bind a given client with the master server in thenetwork of connected clients. Optionally, more than one instance of theclient software program can be launched per computing device/computingserver, including the master server 5200. Optionally, the master serverprogram 5205 enables, via a user interface, a user/operator to specify adata communication port (e.g., a TCP/IP port) in which theassociated/bound master server and client programs can communicate overa local-area or wide-area data network. Optionally, the master serverprogram 5205 enables, via a user interface, a user/operator to refreshthe list of connected clients by, for example, broadcasting a message tothe cluster clients and waiting for their response.

Optionally, the master server program 5205 provides certain clustersynchronization processes to ensure the multi-user VR session isreliably and timely synchronized across all clients. If for example, aclient were to fall behind with respect to rendering or enable renderingin a multi-user session, the VR experience for the lagging client, andother participants viewing the client, might seem sub-optimal,unrealistic and/or sluggish. These processes optionally include but arenot limited to: sync frames, sync draw, and sync resources. Optionally,if the sync frames process is enabled via a user interface of the mastercluster 5205, each client computing device/server blocks (e.g. delays aperiod of time, delays until a condition is satisfied, delays until asignal is received, delays until a routine completes, delays until amessage is received, etc.) at the end of each frame until all othercomputing devices/servers finish processing their updates for that frame(as determined by the master server program 5205 by, for example,receiving a frame update complete message from each client or itsequivalent (e.g., a function call unblocks)). For example, in a blendscreen simulation configuration, it may be the case that a client wouldprefer a drop in frame rate rather than ever seeing a discrepancy (e.g.,a screen tear) across blended or butted displays. Optionally, if syncdraw is enabled and sync frames is enabled, in addition to blockinguntil all updates are complete, the client computing device/servers5215, 5225 will optionally also block until all other client computingdevice/servers 5215, 5225 finish rendering their respective frame.Optionally, if the sync resources function is enabled, all resources ina script/executable code folder are automatically synchronized with theconnected clients when running a VR simulation/session. Optionally, whenthe resource syncing function is enabled, a resource filter can beconfigured/programmed by the user/operator to block certain file typesfrom being synced from the master with the clients. For example, a usercan exclude log files from being sent to a specified client or allclients (e.g., to reduce network congestion and latencies). Optionally,multiple filters can be configured/programmed to restrict more than onetype of resource (e.g., models, textures, etc.) synchronization withconnected clients.

Optionally, the master server program 5205 executing on the masterserver provides certain cluster session timer controls to ensure themulti-user VR session is adequately synchronized across all clients. Asynchronization timeout or series of synchronization timeout events maycause an operator or user of the VR session to halt the VR session andtake corrective action to repair the troublesome client(s).

Optionally, the master server program 5205 enables a user/operator tochange a default time period (e.g., 100 milliseconds, 500 ms, 1 sec,etc.) the cluster master and client program will wait after a timeoutoccurs before attempting to establish a data connection (e.g., a TCP/IPdata connection) to attempt synchronization. Optionally, in the event ofa timeout, the cluster master 5205 and/or client program 5215, 5225 willautomatically retry synchronization a configurable number of automaticretries before determining that the problem that caused the timeout isnot correctible without further action. Optionally, a failed retryand/or a failed connection attempt is reported to the user/operator ofthe client and/or master server as an alarm/error (e.g., via a userinterface displayed on a display and/or via an audible alert) and/orlogged as an error to an error log file.

Optionally, the master server program 5205 enables a user/operator tochange, via a user interface, a default time period which the clustermaster 5205 and client program 5215, 5225 will wait to finishsending/receiving messages for each frame in a VR session (e.g., 100milliseconds, 500 ms, 1 sec, etc.). Optionally, in the event messagesending and/or receiving for a frame exceeds a specified default timelimit, a failed frame alarm/error may be reported to the user/operatorof the client and/or master server (e.g., via a user interface displayedon a display and/or via an audible alert) and/or logged as an error toan error log file.

Optionally, the master server program 5205 enables a user/operator tochange a default time period the cluster master 5205 and client 5215,5225 programs will wait for synchronization to complete at the end ofeach frame (e.g., 100 milliseconds, 500 ms, 1 sec, etc.). Optionally, inthe event the synchronization for a frame exceeds the specified defaulttime limit, a failed frame alarm/error may be reported to theuser/operator of the client and/or master server (e.g., via a userinterface displayed on a display and/or via an audible alert) and/orlogged as an error to an error log file.

Optionally, during the initialization and/or connection process, anexchange of resource types and/or hardware component types andcapabilities occur between the master server program 5205 and associatedclients programs 5215, 5225. For example, the resource and hardwareexchange can include the manufacturer and model of the client headmounted display, the pixel resolution of the client display, the makeand model of the graphic processing unit, the make and model of a handheld wand used in association with a client, etc.

Virtual Reality Rendering Startup

In this example, after the initialization and connection processes arecomplete, the master initiates a second command or sequence of commandsto cause the clients to load the executable code of the VR scene in therender engine of each client computing server 5210, 5200. Optionally,any problems and/or detected errors in loading the VR scene into therender engine are reported to the first user 5150 and second user 5155via the user's head mounted display 5170 and 5175, respectively, and/ora separate console/monitor of the client computing server 5210, 5200.Optionally, any detected problems and/or detected errors are mirrored toand/or exclusively provided to the master server program 5205.Optionally, any detected errors are logged to the master server's datastore 5202 and/or displayed via a console/monitor to an operator of themaster server 5200. Optionally, the master server program 5205 takescertain automatic action to repair detected errors, including forexample, reloading the render engine of the client 5210, 5200,optionally, with certain modified parameters. Optionally, the masterserver program 5205 waits for a repair action to be initiated by theoperator. Optionally, each client in the network of clientsindependently loads their respective render engine. After each client'srender engine is loaded, the multi-user simulation is ready to begin.

Virtual Reality Simulation

The cluster system is used to share the rendering of a scene acrossmultiple image generators and/or display screens. The cluster system,and in particular the master server software 5205, manages theviewpoint, or the rendered view, of each connected client, for example,by sending a sync command with certain position and orientation matricesof one or more avatars associated with the VR scene. By having themaster perform certain computational tasks as described herein, theclient's computational load is reduced enabling the clients to run moreefficiently. Advantageously, as previously described herein, therendered view is optionally further converted by the master serverprogram 5205 to a set of commands/formats compatible with the hardwareof a given client based on an identification of the clienthardware/software obtained during initialization or registration (e.g.,the pixel resolution of the client display, the display type, etc.).Further, because the cluster system as a whole performs the motioncapture tracking, image processing, position calculations, etc., andconverts the received input into perspectively correct renderings foreach client's view, the VR program creator is divorced from these lowlevel technicalities and instead interfaces with an abstracted view ofthe cluster system, and can thus focus on other aspects of thesimulation (e.g., the simulation model, training tasks to be achieved bythe simulation, the simulation scripting, etc.).

As previously described, the cluster system may optionally cause therendering of certain aspects of the VR simulation to occur on thecomputing device/server of the client (e.g., computing in a localcontext), independent from, and not immediately synchronized with otherparticipants in the VR session. This local context positiondetermination and image rendering will hereinafter be referred to as“privileged.” Privileged operations and/or rendering enables the clustersystem to provide immediate display/action feedback to the user (thiscan also be characterized as “high frequency” feedback) with little orno human perceptible latency (e.g., less than 10 ms, less than 5 ms,less than 1 ms), thereby enabling the simulation to feel as lifelike aspossible.

An example of an action that may be treated as privileged is a rotationof a user's viewpoint display (e.g., a user wearing a head mounteddisplay, configured with PPT emitters, looking to the right, left, up,down, etc.). It has been determined through research that if latenciesin a visual rendering of a scene in response to a head movement exceedapproximately 10 milliseconds, then the rendering of the virtual sceneis no longer indistinguishable from a real scene from a participant'sperspective. Depending on the individual and the task at hand, thediscrepancy from a real scene as detected by the human visual system maylead to disorientation or sub-optimal performance. Therefore, in manytypes of simulations this action type may be configured to beprivileged. Another example of an action which may be treated asprivileged is tracked travel (e.g., walking) of a user. In both of theseexample privileged actions, the received tracking andpositioning/orientation change data of the user are processed andrendered substantially immediately in a local context by the clientcomputing device associated with the user.

Notably, the cluster system enables a first party to the system (e.g., auser, a participant, an operator, a content creator, and/or adeveloper/scripting agent, etc., or any combination thereof) todefine/configure/program/designate the system to cause certain aspectsor elements of the VR simulation to be rendered in a privileged manner.Optionally, the cluster system recommends to the first party, via a userinterface displayed on a display, which aspects/elements should beprivileged based at least in part on certain predefined, detected,and/or measured criteria including, for example, a measured latency of anetwork connecting the computing devices and/or tracking cameras thatexceeds a specified threshold, or based at least in part on a measuredprocessing latency of a connected computing device exceeding a thresholdas determined, for example, by having the computing device performcertain computational operations. Optionally in addition or instead, thecluster system recommends to the first party which aspects/elementsshould be privileged based at least in part on the mass/size of amoveable virtual object and certain physics associated with an object(e.g., a large rock on a hill or a key in a lock). Optionally, inaddition or instead, the cluster system recommends to the first partythat certain predefined aspect/elements should be considered privilegedsuch as head tracking, user travel, user limb movement, etc. Optionallyin addition or instead, the cluster system recommends to the first partywhich aspects/elements of the system should be privileged based at leastin part on the identification/detection of certain haptic devices(including, device make and/or model) on initialization wherein,generally, haptic devices are configured as privileged.

Optionally, the cluster system automatically configures for the firstparty certain aspects/elements of the system as privileged or notprivileged based at least in part on one or more criteria describedherein (e.g., measured latencies, measured processing capabilities,physics, haptic device outputs, etc.). For example, if the systemmeasures a network latency between a master computing device 5200 and aslave/client computing device 5210, 5200 that exceeds 10 milliseconds(or other specified threshold) the computing device may be automaticallyconfigured to operate in a privileged manner with respect to certainfunctions/portions of the simulation (e.g., previously identified ascritical or potentially privileged) to thereby provide relativelyreduced latencies.

Optionally, the system may provide a user interface via which the firstparty may define/configure/program/designate the system to cause certainaspects of the VR simulation and/or elements of the simulation to berendered in a privileged manner dependent upon the type of application.The system may receive and act on the information and commands providedvia the user interface. For example, in a multi-user sport trainingsimulation, ball physics may be privileged to each user client. Inanother example, in a flight vehicle simulator certain responses, saycontrol yoke and side-stick, need to be privileged in order to provide alow latency, realistic experience, whereas the attitude and position ofthe virtual flight vehicle do not need to be privileged as some latency(e.g., up to 200 ms or other threshold), will not adversely affect therealism of the virtual reality experience. As described herein, headmovement and travel may optionally be generally configured as privilegedoperations. Limb movements, may be generally configured as privilegedoperations but can also be configured as non-privileged given measurednetwork latencies and computing device processing capabilities.

Taste and smell are not generally rendered visually per se and,therefore, may be categorized as non-privileged in the client-servercluster configuration domain (which may optionally categorize whichaspects may be privileged and which aspects are never to be treated asprivileged). However, with respect to sound in multi-user simulations,there are certain situations in which it may be advantageous for soundto be rendered in a local context or privileged operation. Optionally,sound rendering intended to convey accurate angular and distance cues(e.g., gunshots, car engine and tire sounds, the sound of an objectstriking a surface, a person speaking, a dog barking, a door opening orclosing, footsteps, etc.) may need to be privileged operations (and soare categorized as potentially privileged) because they are susceptibleto latency and may be closely synchronized with a corresponding visualrendering. Ambient sound, such as, for example, background noise ormusic with no pinpoint localization needed, can be configured asnon-privileged operations (e.g., processed by a master server or enabledby a communication channel via the master server).

In the cluster system, once a position matrix (which may includeorientation information) of an action (e.g., of a user, part of a user,or object, etc.) is derived from the received tracking data by a client5215, 5225 for a privileged action, the position matrix is sent to themaster server program 5205. Optionally, a client 5215, 5225 performs andtransmits the position and orientation determination for each frame (orset series of frames) in order that the master server program 5205 hasas much information coming in as possible about the privilegedoperation. The position matrix received from the client 5215, 5225,coupled with the position data of the other simulation users (e.g.,previously stored position data or position data determined/derived fromtracking data), enables the master server program 5205 to provide scenerendering information to each client 5215, 5225 wherein each client5215, 5225 is configured to receive over the network scene position andorientation data suitable for local rendering (e.g., by a head mounteddisplay or other display) and wherein the received client renderinginformation includes actions taken by other simulation users in aperspectively correct viewpoint. By way of illustration, an example of aperspectively correct viewpoint is where a virtual wall of thesimulation may obstruct an observer's view of another user when the usertravels, in the simulation, behind the virtual wall. In an illustrativeexample, the master server program 5205 may provide client 5225 positionand orientation for the first user in a data array in response tocertain conditions (e.g., frame event, time out event, set series offrames (e.g., 2, 3, 4, 5 frames, etc.), etc.) for local scene rendering.Similarly, the master server program 5205 may provide client 5215 withposition and orientation for the second user.

Certain other VR actions may be considered and processed by the clustersystem as non-privileged (e.g., remote or not in a local processingcontext). For example, the movement of a tracking wand or the movementof a user's hand may be categorized and treated as non-privileged.Non-privileged user action tracking data (e.g., marker tracking identityand position data received from motion tracking camera(s) or trackingcomputing server 5230) is processed by the master server program 5205and positional scene changes/actions are identified for each connectedclient 5215, 5225 and downloaded (e.g., via a sync operation) to theconnected clients for local context rendering (e.g., wherein each client5215, 5225 is configured to receive over the network scene position andorientation data suitable for local rendering), including theclient/user from which the action originated (as further describedbelow).

In this example, each user is wearing a head mounted display 5170, 5175with an attached PPT emitter (e.g., LED equipped PPT Eyes). Each user5150, 5155 in this example, is also wearing a right-hand glove 5160,5165 configured with finger motion sensors and one or more infraredLEDs. The LED(s) associated with the PPT emitter and glove may each emita unique infrared light pattern/code (e.g., marker Identification (ID))which is recorded by the tracking cameras. The light patterns emitted bythe LEDs uniquely identify a user's head movement, user travel, anduser's right hand movement. The PPT tracking camera output (e.g., eitherPPT X indirect output wherein the output may be processed by, forexample, using PPT studio software using an intermediate computingdevice 5230 (e.g., a dedicated computing device or a shared computingdevice such as the server 5230 or PPT E direct output)) is received bythe master server program 5205 and client programs 5215, 5225. Theoutput from the intermediate computing device 5230 or direct output inthe case of PPT E provides infrared marker identify and space positionof the LED markers. The client programs 5215, 5225 similarly receiveglove 5160, 5165 marker ID and finger movement sensor data.

FIG. 9 and FIG. 10 illustrate a first example process workflow of theexample clustered system discussed above. Process states are listed onthe left and elements of the operating environment of FIGS. 1, 2, and 3are listed across the top.

State 1 of FIG. 9. An instance of the master server program 5205launches on the computing server 5200 (e.g., in response to a launchcommand provided by a system administrator via monitor 5260).

State 2 of FIG. 9. The master server program 5205 generates and providesfor display a user interface on a console/monitor 5260 that identifiesthe network connected computing devices 5210 and 5200 (e.g., usingdifferentiating identifiers, such as an alphanumeric name and/or anicon). The master server program 5205 receives, via the user interface,a selection by the system administrator of an add control, and inresponse, the master server program 5205 creates an instance of a client5215 on the computing device 5210 and binds (e.g., creates aclient-server association/linkage enabling the master to send masterinitiated requests and data over a communication channel and receiveclient responses, client initiated requests, and data or other computerprogramming methods of software program/module association as known tothose skilled in the art of data communications and/or softwaredevelopment) the created client 5215 with the master server program5205. In this state or an earlier or later state the configured clients5215,5225 and master server access configuration information indicatingwhat types of operations associated with the simulation are privilegedand what types of operations are non-privileged.

State 3 of FIG. 9. The actions of State 2 are repeated to create andbind client 5225 on the computing device 5200 with the master serverprogram 5205.

State 4 of FIG. 9. The master server program 5205 initializes thesimulation (e.g., in response to determining that the systemadministrator has selected an initialize control presented via a userinterface of the master server program 5205 presented on console/monitor5260), and downloads, over the network 5250, to each client computingdevice's data store 5212, 5222 executable code and resources to be usedin the simulation. In States 1, 2, 3, or 4 (or in a separate state) thetracking cameras 5310-5340 are also initialized and calibrated (e.g.,with respect to the marker tracking devices used in the simulation). Inaddition, the unique infrared marker IDs (e.g., by each marker emittinga distinctive pattern) of the tracking devices are associated with users5150, 5155 and clients 5215, 5225, respectively.

State 5 of FIG. 9. The simulation session begins (e.g., in response todetecting that the administrator selected a user interface control ofthe master server program 5205 presented on console/monitor 5260initiating a virtual world script on the master server 5205). Inresponse to the control selection, the master server 5200 initiates asimulation in each client 5215, 5225 (e.g., via a synccall/command/instruction).

State 6 of FIG. 10. In response to the VR launch, a display/scene ofRoom 5100 is rendered (or caused to be rendered) in the head mounteddisplay 5170, 5175 of each user 5150 (first user), 5155 (second user)via the client software 5215, 5225. In this example, the first user'sviewpoint/display includes an avatar (e.g., a virtual representation) ofthe second user. The second user's viewpoint/display includes an avatarof the first user.

State 7 of FIG. 10. First user 5150 walks from the front (East side) ofRoom 5100 to the sliding door 5110. The PPT motion tracking cameras5310-5340 (e.g., PPT X cameras) record the first user's 5150 steps,including in this example, the infrared light signal emitted by thetracking markers (e.g., a left marker and a right marker of a PPT Eyestracking device) affixed to the Head Mounted Display (HMD) 5170. In thisexample, a computing device 5230 hosting a motion tracking softwareprogram 5235 (e.g. PPT Studio) performs certain operations on the cameravideo output, including but not limited to, one or more of thefollowing: an analog to digital conversion of the captured video (usingan analog to digital conversion device and/or conversion software),identification of marker patterns and from the marker patternsidentifies each marker in the camera field of view, calculations of twodimensional centroids of marker positions, determine the center pointposition between the two markers of the tracking device (e.g., PPT Eyestracking device), etc.

At least partly in response to accessing the privileged operationconfiguration information, the client 5215 executing on the computingdevice 5210 receives the tracking data from computing device 5230 andperforms certain calculations with respect to the infrared markersassociated with the first user 5150 to create a series of positionmatrices corresponding to the first user movement (e.g., wherein a givenmatrix may express the position/orientation coordinates in space of thefirst user 5150 in 5 degrees of freedom, x, y, z, yaw, roll, or otherdegrees of freedom) and a new viewpoint is rendered (and/or causes a newviewpoint to be rendered) in the client/display 5215, 5170 (e.g., via alocal context privileged rendering operation as previously describedherein).

The process of State 7 may be continually repeated as the first user5150 travels/walks to the sliding door 5110 with the first user's headmounted display 5170 displaying an expanding view of the wall separatingRoom 5100 from Room 5105 and the sliding door 5110 (e.g., a scene of thesimulation). Optionally, PPT E motion tracking cameras are used in placeof or in addition to PPT X cameras, in which case the camera outputincludes marker identification and associated marker position data(including each marker position and/or a center point position betweentwo markers of a two marker tracking device) eliminating the need forthe standalone computing device 5230 to perform the operations.

State 8 of FIG. 10. Once each new first user position matrix isdetermined/generated for the first user in State 7, the new positionmatrix is sent/streamed by the client 5215 over the data network 5250 tothe master server program 5205. Optionally, the master server program5205 receives the new position matrix in response to a serverrequest/calling function.

State 9 of FIG. 10. With respect to the master server program 5205, themaster server program 5205 receives the position matrices from theclient 5215. With each received position matrix corresponding to thefirst user movement, the master server program 5205 generates, based atleast in part on and taking into consideration the current position data(e.g., including head position and body location) of the second user5155, a new orientation/position of the first user avatar (referred tohereinafter as avatar1). This new orientation/position of avatar1 isdownloaded (e.g., via a synchronize call/operation/instruction) from themaster server program 5205 to the second client 5225 (and/or clientcomputing device 5200) for rendering in the head mounted display 5175 ofthe second user 5155. As each synchronized update is received from theserver 5205, the associated avatar1 positioning in the scene is renderedby the client/display 5225, 5175. The second user views in their headmounted display 5175 avatar1 walking towards the sliding door 5110.

As illustrated with respect to States 7-9 of this example, the clustersystem manages the technicalities discussed herein associated with thesubstantially real-time communications between client and server andserver and clients (e.g. real-time in nature but subject to certainprocessing and network transmission delays/latencies). In addition, thisexample illustrates how the cluster system manages/enables the variousavatar renderings to occur in their perspectively correct views for eachuser in the multi-user system, wherein the user's view or viewpoint ofeach avatar/participant in the simulation is based in part on positionalchanges of each avatar/participant in the simulation (optionallyincluding the extremities of each avatar/participant) together withpositional changes from which each user views the simulation. Thus,because the cluster system manages the technicalities, the simulationcontent creator/developers may concentrate their creative efforts on thesimulation application itself rather than having to code/developprograms to deal with such technicalities.

State 10 of FIG. 10. Upon reaching the sliding door 5110, the first user5150 in this example reaches out with the gloved hand to grab a handleon the sliding door 5110. In this example, arm movement is treated as anon-privileged operation (e.g., a remote server operation). The PPTmotion tracking cameras 5310-5340 record the first user's 5150 handreaching towards the handle via the identified infrared light signalemitted by the LED marker(s) attached to, for example, the glove 5160.At least partly in response to accessing the privileged operationconfiguration information, the master server program 5205 executing onthe computing device 5200 receives the glove tracking data fromcomputing device 5230 and performs certain calculations with respect tothe received infrared marker(s) associated with the first user's hand tocreate a series of hand position matrices (e.g., where a given positionmatrix expresses the position coordinates of the first user's hand). Theresulting one or more position matrices are downloaded (e.g., via asynchronize command/operation/instruction) from the master serverprogram 5205 to the clients 5215, 5225.

State 11 of FIG. 10. Upon receipt by the client 5215 of the handtracking position data, a new position of the first user's hand isrendered (or caused to be rendered) in the first user's head mounteddisplay 5170. Upon receipt by the client 5225 of the hand trackingposition data, a new position of the first user's hand in Avatar1 isrendered in the second user's head mounted display 5170. The positioncalculation sequence is repeated by the master server program 5205 andrendering by the clients 5215, 5225 is repeated as the gloved hand ofthe first user reaches for the sliding door handle.

State 12 of FIG. 10. In this example, first user 5150 grasps with thegloved hand the handle on the sliding door 5110. From the received lightemitted from the glove markers of the first user (e.g., the infraredlight recorded by one or more of the cameras 5310-5340), and at leastpartly in response to accessing the privileged operation configurationinformation, the master server 5205 identifies by way of the output fromcomputing device 5230 that the first user's right hand has made contactwith the sliding door handle (e.g., via a position in space of the firstuser's extended right hand) of the simulation (note, in this virtualreality simulation embodiment there is no physical door or handle).

In this example, the master server 5205 instructs (e.g., via anon-privileged master server synchronize command/operation/instructiondirected only to client 5215 using, for example, a masking synchronouscommand/operation/instruction) the client 5215 to create a sense oftouch in the gloved hand of the first user, by for example, causing viaan instruction/operation the inflating of a collection of air pockets inthe glove 5160 on the palm side to create pressure force against theuser's hand. Thus, optionally, touch feedback is processed/caused as anon-privileged operation.

Alternatively or in addition, optionally the cluster system uses thesensors in the glove 5160 (rather than the determined position of theglove hand) of the first user to detect a grasp action by the user (thisexample privileged operation is illustrated in FIG. 10 rather than anon-privileged operation described herein above). At least partly inresponse to accessing the privileged operation configurationinformation, the client 5215, upon detecting the glove grasp (e.g.fingers closing), in a client privileged operation, instructs the gloveto create a sense of touch as previously described herein.

State 13 of FIG. 10. In this example, first user 5150 opens the door5110 with a left to right movement of the gloved hand. The master server5205 generates a series of hand position matrices (e.g., via thereceived tracking data from computing device 5230 as previouslydescribed and at least partly in response to accessing the privilegedoperation configuration information, for example, with respect to State10). A series of new arm position movements are determined anddownloaded (e.g., via a synchronize commands/operations/instructions)from the master server 5205 to the clients 5215, 5225.

State 14 of FIG. 10. Upon receipt by the client 5215 of the handtracking position data, a new position of the first user's hand and thedoor 5110 is rendered (or caused to be rendered) in the first user'shead mounted display 5170. Upon receipt by the client 5225 of the handtracking position data, a new position of the first user's hand inAvatar1 and the door movement 5110 is rendered in the second user's headmounted display 5175. The position calculation sequence is repeated bythe master server program 5205 and rendering by the clients 5215, 5225as the gloved hand of the first user opens the sliding door 5110.

State 15 of FIG. 10. First user 5150 releases the door handle, which inthis example, is treated as a privileged client operation, and so isprocessed locally by the client 5215. At least partly in response toaccessing the privileged operation configuration information, the client5215 upon detecting the handle release via sensors in the glove (e.g.,by detecting the opening of the user's hand), instructs the glove 5160to remove the pressure force (e.g., by deflating the collection of airpockets in glove 5160.)

Optionally, the door handle release action is neither identified norreceived in this example by the master server program 5205, andconsequently, the action is not rendered in the client 5225. Optionally,the cluster system enables one or more clients 5215, 5225 tosend/receive communications between each other. Optionally,client-to-client communication is enabled via the master server program5205. Optionally, client-to-client communication is enabled via userinterface control(s) of the client and is not enabled via the masterserver program 5205. Optionally, the handle release action of thisexample is communicated directly from the client 5215 to the client 5225in a peer-to-peer communication (i.e., not via the master server program5205). Optionally, the receipt of the handle release action by theclient 5225 causes the avatar1's right hand as viewed by the second user5155 to be colored an unnatural skin color (e.g., green) via customizedsoftware of the clients 5215, 5225. Thus, as illustrated in thisexample, the cluster system facilitates/enables client communicationsand through these communications, customized software of the simulationcan be developed/incorporated to provide certain simulation effectsspecific to one or a plurality of clients.

State 16 of FIG. 10. First user 5150 walks through the door opening. ThePPT motion tracking cameras 5310-5340 record the first user's 5150 stepsthrough the simulated door 5110 via the infrared light emitted by thetracking markers affixed to the head mounted display 5170. At leastpartly in response to accessing the privileged operation configurationinformation, the client 5215 executing on the computing device 5210receives from the computing device 5230 tracking data and performscertain calculations with respect to the infrared markers associatedwith the first user 5150 to create a series of position matricescorresponding to the first user movement (e.g., as previously describedwith respect to, for example, State 7), and a new viewpoint is rendered(or caused to be rendered) by the client/display 5215, 5170 for each newposition/orientation matrix (e.g., via a local context privilegedrendering operation as previously described herein). The image capture,processing, and client 5215 rendering sequence is continually repeatedas the first user 5150 walks through the sliding door with the firstuser's head mounted display 5170 displaying a view of the inside of Room5105.

State 17 of FIG. 10. As similarly described with respect to State 8,after each new first user position matrix is determined/calculated inState 16, the new position matrix is sent by the client 5215 over thedata network 5250 to the master server program 5205.

State 18 of FIG. 10. With respect to the master server program 5205, themaster server program 5205 receives the generated position matricescorresponding to the first user from the client 5215. For a givenreceived position matrix from the client 5215, the master server 5200creates, based at least in part on and taking into consideration thecurrent position data (including head position and body location) of thesecond user 5155, a new orientation/position of avatar1. This newavatar1 positioning data is downloaded (e.g., via a synchronizecall/operation/instruction) from the master server program 5205 to thesecond client 5225 (and/or client computing device 5200) for renderingin the head mounted display 5175 of the second user 5155 (see also State9). As each synchronized update is received from the server 5205, theassociated avatar1 positioning in the scene is rendered by theclient/display 5225, 5175. The second user views in their head mounteddisplay 5175 avatar1 walking through the sliding door. In this exampleembodiment, at least a portion of the front part of the avatar's body isconcealed by the wall as the first user enters the second room (Room5105).

State 19 of FIG. 10. When the master server program 5205 determines thatthe first user 5150 has cleared the door passage, the master serverprogram 5205 simulates the door 5110 closing behind the first user. Themaster server program 5205 determines the first user 5150 has clearedthe door passage based on the at least in part on the received positionmatrices from the client 5215 or, optionally, by independentlyidentifying first user 5150 marker data from the tracking data output ofthe computing device 5230. The master server program 5205 downloads(e.g., via a synchronize call/operation/instruction) to the client 5225an update to the VR scene wherein the sliding door closes and/orprovides the client with instructions to close the door in the VR scene.The client/display 5225, 5175 renders (or causes the rendering) a viewof the door closing scene(s) behind the avatar1 in the viewpoint of thesecond user's head mounted display 5175.

Thus, as described above, an example clustered system provides amulti-user virtual reality simulation via a client/server architecture.Advantageously, the client/server architecture enables timely viewing ofcertain privileged user actions and server-based synchronization ofnon-privileged user actions to create a lifelike simulation experience.In addition, many of the technical details of rendering avatarrepresentations of simulation participant in the viewpoint of each useris derived by the server and made available to the clients in a formatcompatible with the hardware used by each participant.

The methods and processes described herein may have fewer or additionalsteps or states and the steps or states may be performed in a differentorder. Not all steps or states need to be reached. The methods andprocesses described herein may be embodied in, and fully or partiallyautomated via, software code modules executed by one or more generalpurpose computers. The code modules may be stored in any type ofcomputer-readable medium or other computer storage device. Some or allof the methods may alternatively be embodied in whole or in part incomputer hardware. The systems described herein may optionally includedisplays, user input devices (e.g., touchscreen, keyboard, mouse, voicerecognition, etc.), network interfaces, etc.

The results of the disclosed methods may be stored in any type ofcomputer data repository, such as relational databases and flat filesystems that use volatile and/or non-volatile memory (e.g., magneticdisk storage, optical storage, EEPROM and/or solid state RAM).

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. The described functionality can beimplemented in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

Moreover, the various illustrative logical blocks and modules describedin connection with the embodiments disclosed herein can be implementedor performed by a machine, such as a general purpose processor device, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. A general purpose processor device can be amicroprocessor, but in the alternative, the processor device can be acontroller, microcontroller, or state machine, combinations of the same,or the like. A processor device can include electrical circuitryconfigured to process computer-executable instructions. In anotherembodiment, a processor device includes an FPGA or other programmabledevice that performs logic operations without processingcomputer-executable instructions. A processor device can also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Although described herein primarily with respect todigital technology, a processor device may also include primarily analogcomponents. A computing environment can include any type of computersystem, including, but not limited to, a computer system based on amicroprocessor, a mainframe computer, a digital signal processor, aportable computing device, a device controller, or a computationalengine within an appliance, to name a few.

The elements of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processordevice, or in a combination of the two. A software module can reside inRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, a CD-ROM, or any other form of anon-transitory computer-readable storage medium. An exemplary storagemedium can be coupled to the processor device such that the processordevice can read information from, and write information to, the storagemedium. In the alternative, the storage medium can be integral to theprocessor device. The processor device and the storage medium can residein an ASIC. The ASIC can reside in a user terminal. In the alternative,the processor device and the storage medium can reside as discretecomponents in a user terminal.

Conditional language used herein, such as, among others, “can,” “may,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without other input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

Disjunctive language such as the phrase “at least one of X, Y, Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z).Thus, such disjunctive language is not generally intended to, and shouldnot, imply that certain embodiments require at least one of X, at leastone of Y, or at least one of Z to each be present.

While the phrase “click” may be used with respect to a user selecting acontrol, menu selection, or the like, other user inputs may be used,such as voice commands, text entry, gestures, etc. User inputs may, byway of example, be provided via an interface, such as via text fields,wherein a user enters text, and/or via a menu selection (e.g., a dropdown menu, a list or other arrangement via which the user can check viaa check box or otherwise make a selection or selections, a group ofindividually selectable icons, etc.). When the user provides an input oractivates a control, a corresponding computing system may perform thecorresponding operation. Some or all of the data, inputs andinstructions provided by a user may optionally be stored in a systemdata store (e.g., a database), from which the system may access andretrieve such data, inputs, and instructions. The notifications and userinterfaces described herein may be provided via a Web page, a dedicatedor non-dedicated phone application, computer application, a shortmessaging service message (e.g., SMS, MMS, etc.), instant messaging,email, push notification, audibly, and/or otherwise.

The user terminals and other computing devices described herein may bein the form of a mobile communication device (e.g., a cell phone),laptop, tablet computer, interactive television, game console, mediastreaming device, head-wearable display, networked watch, etc. The userterminals may optionally include displays, user input devices (e.g.,touchscreen, keyboard, mouse, voice recognition, etc.), networkinterfaces, etc.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As can berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. The scope of certain embodiments disclosed herein is indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A cluster system, comprising: a computing deviceconfigured as a master computing device; a plurality of motion trackingcameras, wherein the plurality of motion tracking cameras is configuredto reside in a physical space coincident with a first user and a seconduser, the plurality of motion tracking cameras configured to detectinfrared light associated with the first user and infrared lightassociated with the second user, and to communicate position dataderived from detected infrared light associated with the first user anddetected infrared light associated with the second user to the mastercomputing device; non-transitory media storing instructions readable bythe cluster system, that when executed by the cluster system, cause thecluster system to: provide a user interface via which the first userand/or an administrator may designate at least one element of a scene asa gaze direction condition for which an indication is to be rendered;access configuration information comprising information indicating whattypes of operations are privileged and what types of operations arenon-privileged; receive, at the master computing device from one or moreof the plurality of motion tracking cameras, position data correspondingto a position of the first user in the physical space and position datacorresponding to a position of the second user in the physical space;cause, by the master computing device, an image corresponding to thefirst user to be rendered at a first virtual position in a displaydevice associated with the second user, wherein the first virtualposition is determined at least in part on the position datacorresponding to the position of the first user in the physical spaceand on a determined viewpoint of the second user; cause, by the mastercomputing device, an image corresponding to the second user to berendered at a second virtual position in a display device associatedwith the first user, wherein the second virtual position is based atleast in part on the position data corresponding to the position of thesecond user in the physical space and on a determined viewpoint of thefirst user; receive at the master computing device, eye-trackingposition data associated with the second user; and determine, based atleast in part on the received eye-tracking position data associated withthe second user and on the determined viewpoint of the second user, agaze direction of the second user; and cause the indication to berendered in the display device associated with the first user at leastpartly in response to determining that the gaze direction of the seconduser satisfies the gaze direction condition designated via the userinterface.
 2. The cluster system of claim 1, wherein the cluster systemis further configured to: cause a second indication to be rendered inthe display device associated with the first user at least partly inresponse to determining that the gaze direction of the second usersatisfies at least a second condition.
 3. The cluster system of claim 1,wherein the cluster system is further configured to: cause theindication to be rendered in the display device associated with thefirst user at least partly in response to determining that the gazedirection of the second user is toward a first element of the scene. 4.The cluster system of claim 1, wherein the indication comprises a colorto be rendered in the display device associated with the first user. 5.The cluster system of claim 1, wherein the cluster system is furtherconfigured to: cause a representative depiction of eyes of the seconduser to be rendered in the display device associated with the firstuser.
 6. The cluster system of claim 1, wherein the cluster system isfurther configured to; determine the viewpoint of the second user usinginfrared light from a plurality of infrared markers associated with thedisplay device associated with the second user.
 7. The cluster system ofclaim 1, wherein the cluster system further comprises: a computingdevice associated with the second user configured as a client computingdevice of the master computing device; wherein the client computingdevice is further configured to: receive image data from a cameraassociated with the display device associated with the second user;derive, based at least in part on the image data received from thecamera associated with the display device associated with the seconduser, the eye-tracking position data associated with the second user;and transmit the eye-tracking position data associated with the seconduser to the master computing device.
 8. The cluster system of claim 1,wherein the plurality of motion tracking cameras are configured toderive an identity of an infrared marker from a detected repetitivepattern of infrared light pulses.
 9. The cluster system of claim 1,wherein the cluster system is further configured to: provide via asecond user interface configured to enable the first user and/or theadministrator to indicate what types of operations are privileged andwhat types of operations are non-privileged.
 10. The cluster system ofclaim 1, wherein the indication is not rendered in the display deviceassociated with the second user.
 11. A cluster system, comprising: acomputing device configured as a master computing device; a plurality ofmotion tracking cameras, wherein the plurality of motion trackingcameras is configured to reside in a physical space coincident with afirst user and a second user, the plurality of motion tracking camerasconfigured to detect infrared light associated with the first user andinfrared light associated with the second user, and to communicateposition data derived from detected infrared light associated with thefirst user and detected infrared light associated with the second userto the master computing device; non-transitory media storinginstructions readable by the cluster system, that when executed by thecluster system, cause the cluster system to: access configurationinformation comprising information indicating what types of operationsare privileged and what types of operations are non-privileged; receive,at the master computing device from one or more of the plurality ofmotion tracking cameras, position data corresponding to a position ofthe first user in the physical space and position data corresponding toa position of the second user in the physical space; cause, by themaster computing device, an image corresponding to the first user to berendered at a first virtual position in a display device associated withthe second user, wherein the first virtual position is determined atleast in part on the position data corresponding to the position of thefirst user in the physical space and on a determined viewpoint of thesecond user; cause, by the master computing device, an imagecorresponding to the second user to be rendered at a second virtualposition in a display device associated with the first user, wherein thesecond virtual position is based at least in part on the position datacorresponding to the position of the second user in the physical spaceand on a determined viewpoint of the first user: receive at the mastercomputing device, eye-tracking position data associated with the seconduser; determine, based at least in part on the received eye-trackingposition data associated with the second user and on the determinedviewpoint of the second user, a gaze direction of the second user; andcause an indication to be rendered in the display device associated withthe first user at least partly in response to determining that the gazedirection of the second user is not toward a first element of a scene.12. A cluster system, comprising: a computing device configured as amaster computing device; a plurality of motion tracking cameras, whereinthe plurality of motion tracking cameras is configured to reside in aphysical space coincident with a first user and a second user, theplurality of motion tracking cameras configured to detect infrared lightassociated with the first user and infrared light associated with thesecond user, and to communicate position data derived from detectedinfrared light associated with the first user and detected infraredlight associated with the second user to the master computing device;non-transitory media storing instructions readable by the clustersystem, that when executed by the cluster system, cause the clustersystem to: access configuration information comprising informationindicating what types of operations are privileged and what types ofoperations are non-privileged; receive, at the master computing devicefrom one or more of the plurality of motion tracking cameras, positiondata corresponding to a position of the first user in the physical spaceand position data corresponding to a position of the second user in thephysical space; cause, by the master computing device, an imagecorresponding to the first user to be rendered at a first virtualposition in a display device associated with the second user, whereinthe first virtual position is determined at least in part on theposition data corresponding to the position of the first user in thephysical space and on a determined viewpoint of the second user; cause,by the master computing device, an image corresponding to the seconduser to be rendered at a second virtual position in a display deviceassociated with the first user, wherein the second virtual position isbased at least in part on the position data corresponding to theposition of the second user in the physical space and on a determinedviewpoint of the first user; receive at the master computing device,eye-tracking position data associated with the second user; determine,based at least in part on the received eye-tracking position dataassociated with the second user and on the determined viewpoint of thesecond user, a gaze direction of the second user; and at least partly inresponse to a determination that the gaze of the second user is directedat the first user, cause a depiction of the second user's eyes to berendered in the display device associated with the first user, whereinthe depiction of the second user's eyes depicts the second user's eyesas being aligned with the first user's eyes.
 13. A cluster system,comprising: a computing device configured as a master computing device;a plurality of motion tracking cameras, wherein the plurality of motiontracking cameras is configured to reside in a physical space coincidentwith a first user and a second user, the plurality of motion trackingcameras configured to detect infrared light associated with the firstuser and infrared light associated with the second user, and tocommunicate position data derived from detected infrared lightassociated with the first user and detected infrared light associatedwith the second user to the master computing device; non-transitorymedia storing instructions readable by the cluster system, that whenexecuted by the cluster system, cause the cluster system to: accessconfiguration information comprising information indicating what typesof operations are privileged and what types of operations arenon-privileged; receive, at the master computing device from one or moreof the plurality of motion tracking cameras, position data correspondingto a position of the first user in the physical space and position datacorresponding to a position of the second user in the physical space;cause, by the master computing device, an image corresponding to thefirst user to be rendered at a first virtual position in a displaydevice associated with the second user, wherein the first virtualposition is determined at least in part on the position datacorresponding to the position of the first user in the physical spaceand on a determined viewpoint of the second user; cause, by the mastercomputing device, an image corresponding to the second user to berendered at a second virtual position in a display device associatedwith the first user, wherein the second virtual position is based atleast in part on the position data corresponding to the position of thesecond user in the physical space and on a determined viewpoint of thefirst user; receive at the master computing device, eye-trackingposition data associated with the second user; determine, based at leastin part on the received eye-tracking position data associated with thesecond user and on the determined viewpoint of the second user, a gazedirection of the second user; and wherein the second user comprises aplurality of users, and wherein the cluster system is further configuredto cause an indication to be rendered in the display device associatedwith the first user at least partly in response to determining that thegaze direction of at least one user of the plurality of users satisfiesone or more conditions, and wherein the rendered indication includes anidentifier associated with the at least one user whose gaze is causingthe one or more conditions to be satisfied.
 14. A cluster system,comprising: a computing device configured as a master computing device;a plurality of motion tracking cameras, wherein the plurality of motiontracking cameras is configured to reside in a physical space coincidentwith a first user and a second user, the plurality of motion trackingcameras configured to detect infrared light associated with the firstuser and infrared light associated with the second user, and tocommunicate position data derived from detected infrared lightassociated with the first user and detected infrared light associatedwith the second user to the master computing device; non-transitorymedia storing instructions readable by the cluster system, that whenexecuted by the cluster system, cause the cluster system to: accessconfiguration information comprising information indicating what typesof operations are privileged and what types of operations arenon-privileged; receive, at the master computing device from one or moreof the plurality of motion tracking cameras, position data correspondingto a position of the first user in the physical space and position datacorresponding to a position of the second user in the physical space;cause, by the master computing device, an image corresponding to thefirst user to be rendered at a first virtual position in a displaydevice associated with the second user, wherein the first virtualposition is determined at least in part on the position datacorresponding to the position of the first user in the physical spaceand on a determined viewpoint of the second user; cause, by the mastercomputing device, an image corresponding to the second user to berendered at a second virtual position in a display device associatedwith the first user, wherein the second virtual position is based atleast in part on the position data corresponding to the position of thesecond user in the physical space and on a determined viewpoint of thefirst user; receive at the master computing device, eye-trackingposition data associated with the second user; determine, based at leastin part on the received eye-tracking position data associated with thesecond user and on the determined viewpoint of the second user, a gazedirection of the second user; and wherein the second user comprises aplurality of users, and wherein the cluster system is further configuredto cause an indication to be rendered in the display device associatedwith the first user at least partly in response to determining that atleast a threshold number of users in the plurality of users have adetermined gaze direction that satisfies one or more conditions, thethreshold number greater than one.
 15. A cluster system, comprising: acomputing device; non-transitory media storing instructions readable bythe cluster system, that when executed by the cluster system, cause thecluster system to: receive, from a plurality of motion tracking cameras,position data for a first user and a second user in one or more physicalspaces, the position data for the first user and the second user derivedfrom detected infrared light associated respectively with the first userand the second user; access configuration information comprisinginformation indicating what types of operations are privileged and whattypes of operations are non-privileged; cause an image corresponding tothe first user to be rendered at a first virtual position in a displaydevice associated with the second user, wherein the first virtualposition is determined based at least in part on received position datafor the first user in a first physical space and on a determinedviewpoint of the second user; cause an image corresponding to the seconduser to be rendered at a second virtual position in a display deviceassociated with the first user, wherein the second virtual position isdetermined based at least in part on received position data for thesecond user in a second physical space and on a determined viewpoint ofthe first user; receive image data from a camera associated with thedisplay device associated with the second user; determine, based atleast in part on the received image data from the camera associated withthe display device associated with the second user and on the determinedviewpoint of the second user, a gaze direction of the second user; andcause an indication to be rendered in the display device associated withthe first user at least partly in response to determining that the gazedirection of the second user is not toward a first element of a scene.16. The cluster system of claim 15, wherein the cluster system isfurther configured to: cause a second indication to be rendered in thedisplay device associated with the first user at least partly inresponse to determining that the gaze direction of the second usersatisfies one or more conditions.
 17. The cluster system of claim 15,wherein the first physical space and the second physical space are in asame room.
 18. A cluster system, comprising: a computing device;non-transitory media storing instructions readable by the clustersystem, that when executed by the cluster system, cause the clustersystem to: provide a user interface via which a first user and/or anadministrator may designate at least one element of a scene as a gazedirection condition for which an indication is to be rendered; receive,from a plurality of motion tracking cameras, position data for the firstuser and a second user in one or more physical spaces, the position datafor the first user and the second user derived from detected infraredlight associated respectively with the first user and the second user;access configuration information comprising information indicating whattypes of operations are privileged and what types of operations arenon-privileged; cause an image corresponding to the first user to berendered at a first virtual position in a display device associated withthe second user, wherein the first virtual position is determined basedat least in part on received position data for the first user in a firstphysical space and on a determined viewpoint of the second user; causean image corresponding to the second user to be rendered at a secondvirtual position in a display device associated with the first user,wherein the second virtual position is determined based at least in parton received position data for the second user in a second physical spaceand on a determined viewpoint of the first user; receive image data froma camera associated with the display device associated with the seconduser; determine, based at least in part on the received image data fromthe camera associated with the display device associated with the seconduser and on the determined viewpoint of the second user, a gazedirection of the second user; and cause the indication to be rendered inthe display device associated with the first user at least partly inresponse to determining that the gaze direction of the second usersatisfies the gaze direction condition designated via the userinterface.
 19. A cluster system, comprising: a computing device;non-transitory media storing instructions readable by the clustersystem, that when executed by the cluster system, cause the clustersystem to: receive, from a plurality of motion tracking cameras,position data for a first user and a second user in one or more physicalspaces, the position data for the first user and the second user derivedfrom detected infrared light associated respectively with the first userand the second user; access configuration information comprisinginformation indicating what types of operations are privileged and whattypes of operations are non-privileged; cause an image corresponding tothe first user to be rendered at a first virtual position in a displaydevice associated with the second user, wherein the first virtualposition is determined based at least in part on received position datafor the first user in a first physical space and on a determinedviewpoint of the second user; cause an image corresponding to the seconduser to be rendered at a second virtual position in a display deviceassociated with the first user, wherein the second virtual position isdetermined based at least in part on received position data for thesecond user in a second physical space and on a determined viewpoint ofthe first user; receive image data from a camera associated with thedisplay device associated with the second user; determine, based atleast in part on the received image data from the camera associated withthe display device associated with the second user and on the determinedviewpoint of the second user, a gaze direction of the second user; andat least partly in response to a determination that the gaze of thesecond user is directed at the first user, cause a depiction of thesecond user's eyes to be rendered in the display device associated withthe first user, wherein the depiction of the second user's eyes depictsthe second user's eyes as being aligned with the first user's eyes. 20.A cluster system, comprising: a computing device; non-transitory mediastoring instructions readable by the cluster system, that when executedby the cluster system, cause the cluster system to: receive, from aplurality of motion tracking cameras, position data for a first user anda second user in one or more physical spaces, the position data for thefirst user and the second user derived from detected infrared lightassociated respectively with the first user and the second user; accessconfiguration information comprising information indicating what typesof operations are privileged and what types of operations arenon-privileged; cause an image corresponding to the first user to berendered at a first virtual position in a display device associated withthe second user, wherein the first virtual position is determined basedat least in part on received position data for the first user in a firstphysical space and on a determined viewpoint of the second user; causean image corresponding to the second user to be rendered at a secondvirtual position in a display device associated with the first user,wherein the second virtual position is determined based at least in parton received position data for the second user in a second physical spaceand on a determined viewpoint of the first user; receive image data froma camera associated with the display device associated with the seconduser; determine, based at least in part on the received image data fromthe camera associated with the display device associated with the seconduser and on the determined viewpoint of the second user, a gazedirection of the second user; and wherein the second user comprises aplurality of users, and wherein the cluster system is further configuredto cause an indication to be rendered in the display device associatedwith the first user at least partly in response to determining that thegaze direction of at least one user of the plurality of users satisfiesone or more conditions, and wherein the rendered indication includes anidentifier associated with the at least one user whose gaze is causingthe one or more conditions to be satisfied.
 21. A cluster system,comprising: a computing device; non-transitory media storinginstructions readable by the cluster system, that when executed by thecluster system, cause the cluster system to: receive, from a pluralityof motion tracking cameras, position data for a first user and a seconduser in one or more physical spaces, the position data for the firstuser and the second user derived from detected infrared light associatedrespectively with the first user and the second user; accessconfiguration information comprising information indicating what typesof operations are privileged and what types of operations arenon-privileged; cause an image corresponding to the first user to berendered at a first virtual position in a display device associated withthe second user, wherein the first virtual position is determined basedat least in part on received position data for the first user in a firstphysical space and on a determined viewpoint of the second user; causean image corresponding to the second user to be rendered at a secondvirtual position in a display device associated with the first user,wherein the second virtual position is determined based at least in parton received position data for the second user in a second physical spaceand on a determined viewpoint of the first user; receive image data froma camera associated with the display device associated with the seconduser; determine, based at least in part on the received image data fromthe camera associated with the display device associated with the seconduser and on the determined viewpoint of the second user, a gazedirection of the second user; and wherein the second user comprises aplurality of users, and wherein the cluster system is further configuredto cause an indication to be rendered in the display device associatedwith the first user at least partly in response to determining that atleast a threshold number of users in the plurality of users have adetermined gaze direction that satisfies one or more conditions, thethreshold number greater than one.